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The Rational Center, for Research In Vocational Education "has 
provided, technical assistance in evaluation* to selected states over 
•the % past two years. This report covers the second fear's work/ 
which was Resigned to assist states in increasing their, ability -to 
evaluate programs and services for special* populations ... The report, 
*alsb provides information on changes in the' total ^valuation system 
of -the participating states which have oo-eurred since) the start of 
the project* 

The project procedure- was/ tp determine the extent td which a state 
was able to provide <vthe necessary information to deycermipe program 
effectiveness in serving/ individuals; within' these special gtabups / 
(disadvantaged , handicapped , , women, minorities, and persons wit:h 
limited English proficiency) identified by the- Education -Amendments 
of 1976. This . set of needed information relates /to the ' ^ 
accessibility of programs, participation of special populafcibn 
individuals, the additional service provided, aija ;the outcomes 
achieved. ^ 

Following the above/cinalysis, each state was assisted , to develop a 
plan for correcting, the identified weakness and to speqify the x aid 
which the National Cehtegr tecfht* could provide./ This publication 
gives a report on this process apd the result/ant improvements made 
in the states. - 



The National Center is particularly indebted to Bill Stevenson, 
Project Director, Marion- Frankenl Research /Specialist, and Graduate 
Research Associate, Eliseo Ponce'. Significant contributions to the 
project were also made'By M. L. McCaslin, /Associate Director, and 
F. L. McKinney,' Prbgram Director of * the Evaluation, and Policy 
..Division, where the project .was conducted/. *. \ »'\ * - * 

* ' ^ / i" ' * 

Recognition and appreciation are extended to the evaluation 
coordinators In each state who were the main contacts for the 
technical assistance ±eanw Ernest ^Neasham and William Morris in , 
California, Betty Schmidt in Connecticut, Lloyd Lawspn, Robert 
v Ferry,* arid James Harris in Colorado,, and Douglas Patterson and his 
staff in Alabama were most helpful in assisting thie project staff 
Jihile in /the states. Other staff in -each of the states freely 
shared their knowledge and expertise With the project team. 
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Credit is also given to the following reviewers of the draft copy ' 
of this repart: Jesse Clemmons, North Carolina; Charles Shubat, 
Minnesota; and Nancy Lust of the National Center . staff . 

Finally , a special note of appreciation is extended tx£ Nancy Powell 
Project Secretary, and 'Marilyn Orlando, Division Secretary, for f' 
their assistance and to Sharon Pinkham who edited the final ^ 
document. > 



Robert E. Taylor 
Executive Director * 
The National Center for 

Research in\ocational > 

Education 
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CHAPTER ijt 
INTRODUCTION 



Thi6 chapter presents the importance, objectives, arid* . 
methodology of the projdct,* It also discusses the development 
of the assessment instrument which was used to rate the states 1 
ability to ^evaluate programs and services for special 
populations in vocational education, « 
• • % i 

«« * IH 

Importance of , the Project 

The Education Amendments of '1976 mandated that^tate 
boards evaluate, among other things, results of additional - 
services that Stages provide to specialpopulations in, terms of 
"planning and operational processes, Xresults of student • 
achievement, and results of student employment success," 1 
Special populations include women, members of minority groups, 
disadvantaged and handicapped 2 persons^ and persons of limited 
English-speaking proficiency, . 

The law has, therefore\ add^d a new dimension to vocation! 
education evaluation, l£ requires states to evaluate specii _ 
populations enrolled in vocational education on the same basi ; s 
as regular 'vocational .students. This means that a fully 
functioning state vocational evaluation ^system should be able 
to compare or Contrast regular and special population students 
on certain criteria, 

i^^^Cent study conducted by the^ leadership development 
function of the National Cen€er ^revealed that state vocational \ 
education 3 directors coOsidered[ evaluation as their number one 
priority. The project staff conducted another informal study 
among state directors**5f vocational education an£ other state 
staff to prioritize the areas df needfed assistance .% Questions 
dealing with special populations represented three of the six 0 
most pressing problems. These three *areas of concern at the 
state level were (1) determination of . special services for 
special populations, (2) follow-up af special populations, and 
(3J determination of needs of special populations,* 





The Natio^i Center began to assist the states^kn solving 
the foregoing problems during 1978-79 through technical 
assistance to selected ^states , During Year % I, the National 
Center for Research in Vocational Education established working 
relations with, four states — California, Maine, Colorado, and 



Alabama. Each of the Vf^our states had indicated a need for 

assistance in evaiual5i*g programs and services- for special 

populations. The results of this effort were previously 1 
^ported by Stevenson et. al. 6 *w 

. ° , i" k 

: * Objectives f 

! > 

The objective of - this project in Year II wa>s to provide 
technical assistance to four states directed toward the 
improvement o£ the evaluation of programs and services for 
special populations. 

The performance objectives in terms of product were tp 
increase the ability of the participating states to 1 



1. 



determine accessibility of programs to special needs 
populations] 



LCcessLb: 
- ) 



2. determine participation of special populations in 
vocational programs;. , * 

3. identify services provided to special 
populations; i \* 

4. <3etermine outcomes, of programs for special . 
.populations. \ * 



primarily 
complish in^ 
objectives. 



Performance objectives in terms ^f 
described what the project team 
achieving the overall objective 
These activities included 

1. analyzing states 1 abil 
services for special p 




programs' and 



2. providing technical assistance* t6 four states on> 
evaluation of programs And services for special 
population; * — 

3. developing a prioritized list of evaluation for 



special populations; 



4. providing strategies for evaluation system 
modification; 

•#> ■ * . < ■ 



m 57 developing procedures for self-analysis of a state 
evaluation system. . 

One of the most important outcomes in the process of " 
providing technical ^assistance is increasing in' the 
participating ♦states 1 knowledge of their ef fectiveness in 
serving special populations* It was hoped that thffs increased 
awareness would" result in the improvement of services and 
programs for these groups and thus effectively provide » - 

occupational education, and training for individuals classified 
as special populations. I| v \ 

i '■ ' r.. 

Given the greater goal of assisting states in the- 
development of a more res^Dnsive evaluation system for 
vocational education , this technical assistance effort was 
viewed as one mechanism for delivering the knowledge, 
experience, and cumulative findings of the National ^Center to 
the^ield* The technical assistance . team can relay information 
and concerns from the sfield back to the National Center .staff . I 
This information could/ include results 6f, &nd,* reactions to 
the products prpdilcecr by the National Center, needs for 
additional research and development, an$ tlje extent to which . « 
the overall goals of the evaluation function of the Evaluation 
and Policy Division are being achieved. 




V 

, Methodology 

The methodology ofi<j£e Technical Assistance Project was 
influenced by the work of Everett* Rogers and ^Floyd Shoemaker* 
especially that on 'dirf usion of innovations. \ Ronald Havelock's 
"The Change Agent's Guide to innovation in Education" proved to 
be very helpful. ' V. 

, A -* - jb - 

The process of providing technical assistance involved <six 
sequential, interlocking phases. These wgre (1) assessing 
needs, t (2) prioritizing needs, (3) identifying alternative 
solutions, (4) choosing solutions , ~( 5) implementation,, and (6) 
evaluating implementation. Figure 1 shows the schematic 
diagram of the procd&s by which the, Technical Assistance 
Project provided technical services to stattes. Figure 2 -shows 
the schedule of activities.^ 

Prior to the actual process »of providing technical « 
assistance was the selection pf cooperating states. Tfce four 
states (California, Colorado, Alabama, and Maine) participating 
in Year I of the project were contacted to determines whether or 



* ■ \ 



Evaluating 
Implementation 



Implementing 
Solutions 



Assessing Needs 




State Vocattonal Iw&luation 
Needs 




\ • 



Choosing Solutions, £ 



Prioritizing Needs 




fr >s 



Identifying Alternative 

Solutions 1 



Figure 1. *\he process of providing technical assistance to .states* , 
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Figured. Gantt chart showing the schedule of activities. 
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not they wished to continue. Maine did not choose to 
participate in Year II of the project. Connecticut was 
selected as a replacement state based on the criteria used in 
Year I 0% the project. Since the emphasis" was on the 
evaluation of programs and services for special needs 
populations, the question of a state's interest in this area «- 
was an additional consideration. r 

\" Memoranda of understandings between the participating 
states -and the National Center were negotiated. These | 
memoranda of- understandings outlined the mutual ^ ^ 

responsibilities of the parties involved in the project. 

Assessing Needs x 

An analyses was made of each evaluation system for special 
needs populations «• • This analysis consisted. of an assessment of 
the state's ability to (1) determine accessibility of programs 
for special populations, (2) determine participation of special 
populations, (3) identify additional services for special 
populations, and (4) measure outcomes of programs and services 
for special populations. To accomplish the foregoing 
objectives, two -basic procedures were employed: fche interview 
and document analysis. 

. ■■ -j 

A structured interview schedule for key s^tate personnel in 
vocational education department was developed from the* items in 
the evaluation matrix showing causes and indicators of program 
effectiveness (see Figure 3). The interview schedule focused, 
on four areas of concern: access, participation, process, and 
outcomes. The interviews were conducted in each state by the 
project director. 

In addition to the interviews, the project - staff asked. for 
documentary evidence regarding the ability of the state . 
vocational education system to evaluate programs and services 
for special populations. These were state enrollment reports, 
enrollment forms, evaluation instruments, follow-up reports, 
follow-up plans, follow-up forms, prbgram evaluation reports, 
policy materials, and local program applications or plans. A 
document analysis was performed * and used as 3 mechanism to 
crosscheck the information gathered from the interview 
schedule. ^ 
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With the infoi^mation gathered during the interviews and 
from the documents, each state was rated on its ^ability to * - 
evaluate programs and additional services for special 
populations by using an assessment instrument which was # * 
developed by the prpject staff and .reviewed by the Evaluation 
and Policy Division. A report of each state- was then ' 
made which described a state's strengths and Weaknesses 
Relative to its ability to evaluate programs 'and additional 
services for spectral populations. Before the report was 
finalized, a draft copy was, made aqd sent to each state for 
reaction. In areas where there were disagreements on the 
ratings, additional evidence was requested before ratings were 
finalized. A final assessment 1 report was then' written iancl 

submitted to each state. ' / ' ' 

* 

Prioritizing Needs ; 4 " / *• 

* * * ' * 

Each state was askecli to respond to ttie assessment report. 
For this purpose, the project director called a conference^ 
consisting of key state vocatibnal education administrators, 
evaluation specialists, and special education staff. The 
project consultant presented a brief summary of the findings. 
This was followed l?y ppen discussion. The process served, to 
(1) enable th^ .project consultant to clarify certain ^spects ^f 
the report upon request of participants and (2) facilitate 
understanding and acceptance of the report. 

When general agreement was v reache& concerning the key 
problem areas, the state vocational education evaluation 
personnel were asked to prioritize their evaluation needs 
giving consideration to fiscal and ^manpower resources. The 
project consultant and ,the state personnel then planned the 
specific technical assistance which would be provided by the 
National Center project staff. Further, timetables for the 
major problem areas w^re mutually agreed upqju 

Identifying Alternative Solutions 

» + 

The project consultant identified feasible alternative 
solutions to the key problem areas for each of the states. In 
the formulation of suggested alternative solutions, the 
following criteria were observed: , 




***** 



' < * , } •■■ - ■ • 

1. Solution's wfere withhTthe manpower and fiscal resources 
of the state. . % 

2. Solutions offered great potential, value to the staJL 
and less potential 1 damage • ' 4 ' ■ ' 

3. Solutions required minimal investment of time and 

effort;; 

» * * 

4. Solutions. of fered rewards in terms of some side 
payments extraneous tp.^the original problem or 
condition. - * 

The process of identifying alternative solutions involved 
face to face informal dialogues to a greater extent than 
formalized conf^rerices. This procedure allowed the client and 
the project consultant to engage in in uninhibited discussion 
of issues involved. Furthermore, enabled them to gain a 
better understanding >of the' other's id^as. In discussions, the 
project consultant strived to maintain objectivity and /played , a 
supportive role in helping the client make decisions. 

' '•' v 

Choosing Solutions 

This- step was an active process in which the clients made 1 
conscious choices about specific solutions to particular 
problems. .This step involved the process of identifying - 
definitive courses of action to talce by state vocational 
education personnel and the process of legitimization. In both 
of these stepp, the- project^ consultant was ^either directly or 
indirectly involved. 

' — * v 

When the state vocational, education personnel h^d defined 

their positions on the priority issues, an implementation plan 
was developed. This included obtaining the approval, where- 
necessary, of proper authorities sjuch as the state director 
and/or the state board for vocational education. The State 
Advisory Council for Vocatipnal Education was alSo informed of 
the project activities by the project consultant. In the 
legitimization prpcess, the project consultant played a 
supportive role to the 3 evaluation personnel. * 

Implementing Solutions 



lfrioni 
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The major ef forfc of the technical assistance staff was to 
help the states implement chosen alternative solutions. The 
^process involved the delivery of sustained assistance ^through 



^on-site visitations , written communications, and telephone 
communications • % - * ; " 

When assistance required a lengthy deliberation with a 
group* of people ,-project^ consultants made site visitations. 
Project staff critiques* and recommendations* concerning pland 
and instruments were written and sent to the states. These 
*were followed with site visitations whenever necessary. Most 
* follow-up activities ,.. .however, were generally done by 
telephone, * 



t 



Evaluating Implementation 



Feedback and evaluation were a continuous process 
throughout the various phases of the technical assistance 
process. The states were encouraged to communicate freely and 
openly with the project director. Pertinent, information with ( 
regard -'to tfce implementation* ptoce^s was regularly jndnitored 
and analyzed. At the end of the -calendar year, a joint review 6 
of accomplishments by state vocational education evaluation 
officials and the project ^ staff was conducted for each . 
cooper at ing s ta te. A~prptrie^^f -eac&^tate 1 ^ accomplishment- wtS^~ 
made, ' 
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CHAPTER II 

THEORETICAL FRAMEWORK: STRATEGIES FOR 
CHANGING VOCATIONAL ^EDUCATION 
-evaluation SYSTEMS KROM TWO - 
, -COMPETING PE RS PE C$TV E S 



In an effort to provide ^information necessary for the 
understanding of social Change process, this chapter presents 
twp major competing theories dealing with social change} namely 
'the conflict theory and the Consensus theory. The majoi* V 
assumptions, their derivation, and their strategies* car) be used 
to change vocational education evaluation'systems in 6rd6r to 
make them' more effective in improving vocational- education.. 

Concensus Perspective _ 

The consensus perspective has its roots; in the 19th century 
and, until recently, has been the domijiaht' conceptual " ■* . 

perspective Ixl sociologyL* The organicism of Cbmte, .Spencer , 

and Durkheim, the work of functional anthropologists like, 
Malinowski and Radclif fe^Brown, and the work of Webster on 
social taxbnomiets helped shape the more modern consensus 
perspective. H x 

" \ * 
_>Cen?ensus, social orcfer, integration, social solidarity, 
equilibrium:- these are the key words in the consensus 
perspective/ Social systems are viewed as being composed of 
different interdependent elements whicU exist, in equilibrium. 
To maintain this equilibrium, integration of ^personality 
systems into th^ cultural system must occur. * Parsons 
postulated two general mechanisms to perform this function: 
tha m^chanisftts of socialization and social control. 

Fundamental, therefore, to the consensus perspective is th£ 
general* notion of social equilibrium and the mechanism that 
integrate^ different levels of social reality to maintain a 
state of * homeostasis." The major requisite is the integration 
among personal, cultural, and social systems. The major 
assumptions of the consensus perspective can be summarized as 
follow^ 
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1. Social reality fexists different „leve£s Of 
organization-— at almiaimum levels the individual ■ _ v 
and cultural (social) . ^ ^ ' \ * • vv ^ ° 

2, Basic needs for survival exist^foir individualsV ?nd 
the creation of soci^l^organizat ion ^'culture) , " 
represents a way of meeting tliese needs ^ 

f 3. Once elaborated , patter^ of social organization b&^e v 
their own needs which are meet by the further 
elaboration of social (cultural) patterns, 

' 1 

4. Hehce, in order to understand why ^^artic(ular sociafl. 
r (cultural)' pattern* exists, it is necessary to know- the 
level or type o£ need Gip requisite/tnat it meets, 




Strategies for Change - 

The consensus strategies are most widely known. The 
strategies employed in such government programs as the * 
extension servicer comipuni/ty development programs, adult 
feducation r and technical assistance are all basedion the 
consensus perspective. The major assumptions ot tfre consensus 
strategies for change are as follows: 

1. The central problem in planned change is increasing 
* productivity and/or efficiency, 

2, Change largely oqcurs through the spread /6f technical 
-knowledge and superior information frOto 1 *he advanced to 

the less advanced areas, , v ^ 

3. The less advanced areas (or backward sector) serCe as 
a brake on the advanced areas, and thus, limit 
progress and development.. > 

4, The major characteristics of the. backward sector vhich 
inhibit overall' progress and development are lack jof 
knowledge, poor* attitudes, .and lack of resources^ 

*•» . - • * * * , 

The major strategy, therefore, for effecting change (from 
the consensus perspective) is best typified by Rogers 1 classic 
strategy for diffusions o^innovationa, 'This section details 
some applications of his |>tinciples to vocational education. 
His prinqiples are outlined briefly as follows: 



1. Provide technical assistance by change agents. . \ 

^ To be most effective, change agents need to be credible. 
Persons in cfcapger of the 5tate evaluation unirts are the most 
likely persons to assume these positions of. leadership for . 
^hanging* evaluation. - Their educational, background , experience , 
and knowledge of unique educational problems witfiin their own- 
states help establish credibility. Their knowledge of who will 
be of as-sistance to thlm; in evaluation efforts at the local 
level will* help assure working with individuals* who are most 
like themselves. 

Change agentfe have the responsiblity to know the law* % 
pertaining to evaluation requirements and the essential 
dhgracteristics of an effective evaluation, system. Knowledge 
of these requirements and' characteristics will assist change, 
agents in providing technical assistance. However, a thorough 
understanding of the change -process from the .-consensus - 
perspective is also necessary. r ^ * 

2. Es"tablishy rapport with and receive a commitment from 

6 clients. / : . * V 

: _\ • . . : . ; : • • , ; - .. ' v - . . 

| n working witfi''§ducational' age -J" 
established with^the individuals with Whom' the change agents*/ 
will be working. -Rapport is based updn . individual personality, 
credibility r and congeniality, and it. results in the ; 
establishment of trus-t between the change agent and the; client. 
> - ' '• ' ■ ' « 

It is best fehat a written agreement between, change agents 
and clients be 'executed/which should specify the expectations 
of the clients anifl the change agents. ^ The ' change agents < must 
be able to obtain information about the clients 1 work 
environment and 1 the way in "which evaluation 'functions in 
relation to- other agency activities. At the same time the 
clients become; aware* of what the change agent will do for them. 

3. Perform a needs assessment based on what clients are 
required to do and what they> are able to do. 

Change agents in this instance are actually performing, an 
evaluative function., They are looking at an agency's - 
capability to secure information required by law* and the 
additional , information required /if clients wish to encoitfpass 
those elements essential to an effective evaluation system. 



■ ' : ' ■ ■ V ^-./ . . . . 

' Such administrative concerns relating to intrar £iid inter 
agency cooperation .and use of /evaluation reporting must be 
addressed by the change agentf. • Credibility of the change' agent 
£lays an important rolein/this endeavor. Successful 
evaluation is the key to this * facet, of credibility. 

,4* Prioritize • the evaluation needs of the agency iji . / 

' cooperation with clients. - . ' • r - 

* v ' v o s v ' _ ' 

»*.."■■ » 

By means of ar> assessment of evaluation capabilities , 
change agents are in a position to advise clients./ As clients 
recognize-def icienclea in their .capabilities to pterform r 
specif fed evaulfction activities* they can make decisions to 
give priority to certain activities. ^ Pr-iorities are based oh; 
requirements, of mandated legislation, sfcafE Availability , and 
ability as Weli as funding. ^ Once" limitations are established L f 
a cooperative effort between change agent '^and client can begin ^ 
,so evaluation activities can be accomplished* X° 

5. Clients in cooperation ^with change agents' need to identify 
alternative solutions* to evaluation prdblenis. 

■ ■ ■ * \ . ' \ \^ , • * 

Evaluation capabilities tend to vary from state to , state as 
well as" aitibng local 'education agencies . The change- agent can 
assist ^clients by helping^ them think through alternative 
solutions to evaluation problems brought to the forefront by 
legislative mandate and the essential characteristics of .an^ 
effective evaluation system. The pros and cons of £ach *■ 
possible alternative solution must be considered before a 
ehoice of solutions to problems is made* . 

6. Choice of solutions to evaluation problem^ is the \ < 
responsibility of the client. I [ ^ 

There are no 'evaluation manuals which can dictate a method 
of evaluation. Manuals devoted to this subject may present 
alternatives/ but a ghoice of solution to any evaluation 
problem rests with the client. This is because- the 
responsibility for evaluation is the client's — -not the change 
agent's. . * . '\ ' 

4 
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7. Clients need to implement changes in the evaluation 

probess. 1 

- ' • . ' ; : * ■ ■ 

Implementing changes in the evaluation process chosen is 
the next step./ This is the test of the chosen solution* 
Implementation will either insure the capability of clients to 
evaluate or will demonstrate that client^* 1 method is inadequate 
or inefficient, . .^ 

8.. Evaluate the implementation of specified evaluation 
procedures.' . . ^ 

Change agents pan be moite objective than the clients who 
have implemented the evaluation process., in any case it is* 
wise to document the amount and type of effort, if any, a 
client, has made in an are^ to be evaluated before the new 
implementation begins. If* this procedure iis followed, it Is 
possible to tell the extent to which improvements have been 
made. \ > . •■■ ' " . r \ * « 



9. Master the arts of . the different techniques for 
implementing innovations. . ■? 

8 



aftd, Mb€as^in< presented 
follows: * 



28 techniques. These are as- 



informative 

1. Printed Information. 

2. Audiovisual Material 

3. Mass Media 

4. Lecture 

5. Symposium 

6. Demonstration . n4 

7. Survey Feedback ^ « 

8. Discussion - •• 

9. Brainstorming 

10. Consultation* ^ 



^ Persuasive 

11. Personal Interview » 

12. Jtole Playing • V 

13. ^Cooperation 0 ' 
14 v Staff Development * 
15; Differentiated Staffing 

16. Involvement in Product Developmenl 

17. Small-Scale Use of the Innova>ti6ir 

18. Competition ' \ 

19. Promotion of the Product 

20. Endorsement^by Authorities 
A.' Recognition of Trial Users 
22. Financial Incentive 

23 • Overstatement d 



Directive 



/ 



24. 
25. 
26. 

27. 
28* 



Deadlines 
Legal Mandates 
Fkit Accompli 

Strategic Replacement of Staff 
Threats of Punishment 



ERLC 
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The Conflict Perspective 

le consensus approach to social .change has been criticized 
pely descriptive*, It does not give 3 satisfactory Recount 
sources ot^ dynamics bf. social change. Further,' the 
theory does not provide a good basi^ for the analysis of 
history f Particularly ito&t dealing with nonotfderly change. t ^ 

/ Consensus theorists View tension and Strain as 
dysf anctidhal--connoting some form of sickness in the system, 
therefore, disruptive . This has been criticized by* conflict *: 1 

theorists because it disregards conflict's positive factors^ 
It centers its attention upon' problems of adjustment rather 
than upon-conf lict, upon social statics rather than upqn spcial 
* dynamics . 

^ Conflict is defined as "any social situation or process in 
jphich two or more entities are linked by at least one form of 
antagonistic psychological relation or 4t least ipne form 
antagonistic interaction . "-Antagonism involves such states as 
"incompa table goals, mutually exclusive interests, emotional 
hostility, dissensus, violent struggle? regulated mutual 
interference, and the like.* 1Z - 

The conflict perspective, contrary to some views, is as old 
«as f unctionalism. It found its inspiration in the works of two 
German ^sociologists, Karl Marx and Georg Simmel. 3 Recent 
conflict theorists Like Ralf Dahrendorf and Lewis Coser have 
contributed to the refinement of the theory* Conflict the- 
orists take the folllowing- assumptions ^regarding conflict; and 
the nature of social change: r 




1. While social relationships -display systematic features, 
these relationships are' rite with conflicting 
interests. \^ 

2. ThiV fact reveals that social systems systematically 

generate conflict. , 

• ■* * . * 

3. Conflict is therefore an inevitable and pervasive 
feature ok social systems. 

4., Such conflict tends to be manifested in the bipolar 
opposition of interests. r 

• 4 ■ j •* ' 




5. Conflict mo|5t frequently occurs over the distribution t 
•of scare resources, most potably power. 

. / 6. Corftlict.is theofnajor sq^Jrce of Change in social 
systems. ' ^ * 

, 0 , . * : ■ ■ ■ +r 

The basic proposition is that conflict often performs basic ' 
preserving fcunctionsi^ It isyhp^ always causes dysfunction; It 
is> often necessary fot the peaceable maintenance of re- 
lationships y 'and it -is a major precursor of social change. . 
Lewis- Coser summarizes tHe six important functions 6f cdnf 1-ict^ 

. * . . " /' • « . ;*> ' ' ; • . * " ; . 

1. Conflict permits internal dissent ion and 
dissatisfaction to rise to the s^irface and enables a 
grbup to restructure itself or deal with , 
dissatisfaction. ■ * - y $: 

2. „ Conflict proVides the emerg^ce of new Sorms of v' 
Appropriate behavior by surf lacing shortcomings. 1 

3. Conflict provides means of ascertaining the strength of 
current power structures. . 

4; (Conflict works to strengthen the boundaries between ' % 
groups'* bringing out their distinctiveness. 

5. Conflict^ creates bond between loosely structured k 
groups , unifying dissent^and unrelated elements. ' '* ;■ 

i." ' 

6. Conflict works as a stimulus -.to reduce stagnation, v 
Conflict may 'alter society. . / 



Strategies for Change 



• The use of conflict as a strategy for change is a fairly • 
new phenomenon among educators although it has had a long 
history in the armed forces and labor unions. The middle class 
orientation of educators has led them to embrace "an 
anti-conflict, anti-violence orientation. 0 . .[and] this has' 
resulted in rule by consensus and conflict avoidance." 1^ 
Conflict is rejected because it is felt that reaching decisions 
thtough consensus and cooperation is the best method to achieve 
social change. . 



Recent events , however , have shown that educators and other 
"peace loving" citizens are willing to use conflict as a 
strategy for- change after consensus strategy fails. This has 
become socially acceptable. The numerous teacher's strikes and 
sit-ins and the protest marches and ^demonstrations employed by 
such groups as the pro^ERA, an{:i-nuclear , pro- or anti-abortion 
attest to this fact. tk number of ground-breaking laws have 
been passed through the use of conflict strategy. 

^In many instances those individuals who resist change the 
most vigorously will , when convinced of the benefits of a » . 
proposed change, become its most enthusiastic proponents. The 
technical assistance project reported in this publication shows 
that in- ?ome cases those state people who were most questioning 
of the team's conclusions and recommendations eventually 
supported and implemented those recommendations most 
effectively. One .way of finding the most effective solution to 
a problem is to create a situation where individuals are caused 
to take both positive and negative positions on an alternative 
solution, and defend those positions vigorously. In such a 
situation attitudes and positions can be changed. .Conflict can 
be used in a positive way to gain support for needed changes. 

The use of conflict situations\hay also be fraught with 
danger for the uSer. If members of the establishment are 
challenged they may feel threatened 'to the extent that they 
seek reprisals against those taking an opposing view even 
though that view may be perfectly justified and reasonable in 
the eyes of the opposition. Also in some instances , it may be 
impossible to control the direction and the extent of a protest 
once it has started. ' "'" ■ 



President Kennedy is quoted .as saying, "The society which 
does not allow peaceful dissent is assuring itself of violent 
dissent." Until recently this country and its institutions 
have operated almost exclusively on a consensus approach to 
problem solution. New tensions and new opportunities are 
created as leaders learn to cope with and use conflict as a 
method to achieve change. This section, therefore, details 
some conflict strategies for change ii\ vocational education. 



1. Identify afeas of corijtlict and use them as* focal points for 
promoting needed change • 



Serious questions are being asked about the outcomes of 
vocational education. jfc segment of the vocational education 
community believes strongly that the traditional outcome of job 




placement ^should continue as the sole criteria for program 
success. Others insist that^many d^her outcomes in addition to 
placement are important and should be considered in determining 
program effectiveness. .A positive change which could result 
frpm this conflict, if properly channeled could be an 
increased consciousness ,^nd concern for product > evaluation in 
vocational education. 

The recenjt differences of opinion as to the proper role of 
vocational 6<9ucation in social change can be used to achieve 
some positive ends. If minorities arid women who now are 
calling for greater equity in ali of education „ can be convinced 
of the contribution vocational educatj^p can make to these 
groups they can help to initiate needed changes. At the same 
time, this."6velr the shoulder look." vocational education is 
receiving cari create increased opportunities iov all students 
needing occupational training. 

2. Organise groups to "establish a creative tension" within* 
the organization. 

In many organizations the research unit is looked at as the 
"burr under the saddle" by the more traditional segments. The 
research unit, .if properly constituted and supported, cari be a 
strong factor for change through its questioning of every 
procedure employed. The more traditional branches can play atj^ 
equally valuable role by challenging and requiring proof of ^ 
worth of each new idea presented by research. The wise 
administrator learns to balance these two conflicting views and 
take the best of both. 

Many vocational programs have fpund strong advocacy in the 
form o% an agressive advisory committee* Employers in the 
community who know the benefits and needs o^vocational 
education can speak with a strong voice to an administration 
tlrat fails to support programs needed by the community. 

3. Increase the involvement of the different sections |>f 
community (parents, special needs, skilledworkers , %tc.) in 
the process of evaluation. - 

These individuals, because of their different background, 
and different perceived needs, will view programs in a 
different light and ask different sets of questions. This will 
inevitability create tension and possible conflict. If in the 
process it causes vocational education to take a new look*at 



itself, some very positive changes can result. Programs and 
teachers can become more open and accepting, students can learn 
to work with those different from themselves, and a/greater 
need can be served. . " ^ L * 

4. Broaden* the representation on evaluation teams. 

* The number of people who can be fully involved in the * 
process of evaluation (stragegy 3) is limited. A much larger 
number of people can toe included on evaluation teams. The 
benefits will generally be the same, possibly in a lesser 
degree, but certainly more widespread. The support of more 
people who better understand the benefits and needs of 
vocational education through this participation can also 6e 
helpful in stimulating change and improvement. i 

•% % 

5. Establish solid alliances with powerful community groups 
such as Parent Teachers Associations and labor units. 

These community groups can be either strong supporters of 
vocational education or its greatest detractors. Usually this 
y depends upon the degree of their knowledge and involvement with 
vocational education. Soliciting support implies a 
responsiveness to needs. These groups will make demands, 
identify problems, and probe for evidence of effectiveness or 
the lack of its. They may create tensions by their demands, 
but improvement can occur through this change. 

6. Learn the art of protest, boycotts, and strikes, and 
employ these when appropriate. 

Most of us cringe at the thought of using the«e tactics to 
bring about changes. However, there are numerous instances 
where this tactic seems the only way to achieve desirable ends. 
The gains of labor, the civil rights-movement, andev^n now the 
human rights struggle arotind the world all have used these 
measures to bring about change. In many instances we can act 
outselves into a new way of thinking more effectively than we 
can think ourselves into a new way of acting. Sometimes 
conflicting action is necessary for change to occijtr in * 
ourselves or in our institutions. 
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CHAPTER III 

A MODEL FOR EFFECTIVE EVALUATION OF 
Y* _ PROGRAMS FOR SERVICES FOR SPECIAL^) PULATIONS 



During the past few years, a number of federally mandated 
requirements have placed additional requirements on state ''V 
evaluation systems. However, some of the data/information 
required for compliance purposes* are also needed for planning 
and program improvement. In view of these requirements, state 
evaluatipn personnel have expressed a need for v a method of 
assessing the adequacy of state "evaluation systems in both 
mandated and planning requirements especially in the evaluation 
of programs ar*d services for specialt populations. 

The activity designed to meet this need was started in the 
first year of this project by making a tentative list of 
essential elements of an effective evaluation system. During 
the first quarter of the second yearof^th t e project, work 
was begun to identify the elements ^fan'tef fective evaluation 
system that would focus* on vocaticmal education programs, for 
special populations. The project. director developed a matrix 
for those identified elements, A review of literature focusjy*j| ; 
on federal legislation led to the identification of ' additiori^if 
elements and factors to be incliided in tke matrix (see 
Appendik B.) This matrix was^ then critiqued by selected state 
vocational education evaluation personnel and by the evaluation 
technical advisory panel and professional staff of the 
Evaluation andHtolicy Division of the National Center for 
Research in Vocational Education. Comments and suggestions* 
were noted *a1nd incorporated into the . present matrix.. 

The matrix, however, t does not show causal relationships. 
Therefore, a causal model was developed to clarify the 
-relationships of the major variables in the matrix. One of the 
objectives of the project staff was to give people iii 
vogational education evaluation/f urther insights into the 
complex process of evaluating programs and services for special, 
^populations* . t r * 

From the matrix, a self-assessment instrument was developed 
to help the states assess the adequacy of th^ir evaluation 
systems to evaluate programs $nd services for Special ~ 



populations* A discussion of the development l>f the instrument 
is presented in the latter. part of this chapter** 



A Matrix for the Evaluation of Vpcational 
v " ^ T" Programs for Special Bppulations " 

6 - 

Clarification of the role ^nd responsibility of evaluation 
and mpre definitive statements of the concepts upon which 
, evaluation is built are required in order to determine the \ 

effectiveness of a state's evaluation system for special 
♦populations apd in order to make heeded improvement* One of 
^the first steps is to "soft* those factors which show 
effectiveness and those factors which are believed to cause it. 
Unless this "sorting" is done as illustrated in the attached 
matrix, the resultant mixing of cause and show factors can 
confuse both determination of quality and identification of 
needed changes* The <3ata and information expressed in tfoe 
matrix demonstrate the factors which must be determined in 
order to evaluate and recommend improvement in programs and 
services for special populations* . 

Figure 1 Shows that the matrix is divided into four 
quadrants—ACCESS t PARTICI PATION , PROCESS , AND OUTCOMES . The 
items oh the right are indicators which show effectiveness and 
may be the basis for an evaluation. The items on the left are .. 
elements within the vocational education system which should be 
considered for change in order to effect Improvement* 

Special elements within each of the major categories of 
the matrix are shpwn in the succeeding pages* 




\ 



Cause 



Effectiveness 



Show 



ACCi^ 4 

1.0 Facility Factors 

.1.1 Architecture and Equipment 

1.2 Site Location, (non-discriminatory) 

1.3 Site Selection (nondiscriminatory) 

1.4 Modification of Physical Plant 

1.5 Comparable Facilities 

1.6 Housing Opportunities 

1.7 "topographical Factors 

2.0 Educational Factors 

2.1 Recruitment 

2.2 Admission Criteria 

2.3 Program Offerings 

2.4 Attitudinal Barriers 

3.0 Societa^Factors 

3.1 Attitudinal Barriers 

3.2 Behavioral Barriers 

3.3 Economic Barriers 



PARTICIPATION 

1.0 Enrollments 

1.1 Disadvantaged $ 

1.2 Handicapped 

1.3 Minorities 

1.4 Limited English-Speaking 

1.5 Sex Designation 

1.6 Age (elderly) ■ 

1.7 Instructional Setting 



PROCESS 

1.0 /'Additional Services" \ 

1.1 Administration Related 

1.2 Guidance and Counseling Related 

1.3 Instruction Related 

1.4 Placement Related 



OUTCOMES 

1.0 Student Achievement 

1.1 Skills 

1.2 Acquisitions in the Affective 
Domain 

2.0 Successful Program Completion 
2:1 Grades 11 and 12 
,2.2 Postseconc[ary 

2.3 Adult 

2.4 Apprenticeship 

3.0 Successful Placement 

3.1 Employed 

3.2 " Unemployed 

3.3 Pursuing Additional Education 

3.4 Status Unknown 

4.0 Successful Employment Oyer Time 

4.1 Duration 

4.2 Promotions 

4.3 Salary Increases 

. 4.4 ? Reaction to Training 



r 



C 



Figure 3. -Matrix for the evaluation of programs and- services 
for special populations . 
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ACCESS 



0 Facility Factors v . X 

. ... . ■. ' ■■ J ■ 

* 1.1 Architectural and equipment 

* \.% Site location (non-discriminatory) 

* 1*3 Site selection (non-discriminatory) 

* 1.4 Modification of physical plant 
• * * 1.5 Comparable facilities 

* 6 Housing opportunities - v . . 

* 1.7 Topographical . * 



0 Educational Factors 
2.1 Recruitment 



* 2.1.1 Public. notification 

* 2.1.2 Promotional efforts 

2.1.3 Identification of potential students in school 
; ; system , \ • 

2.1.4 Identification of potential students outside of 
the school system 

2.1.5 Patent motivation 

2.2 Admission Criteria (Discriminatory practices tq avoid) 

* 2.2.1 Eligibility based on residence 

* 2.2.2 Eligibility based on numerical limits 

* 2.2.3 Eligibility based on student option (race, 

~ national origin and sex designation) 

* 2.2.4 Eligibility based on applicant evaluation • 

* 2.2.5 Eligibility based on language 

* 2.2.6 Access based on employment opportunities 

(handicapped) i" i . 

* 2.2.7 Eligibility based on age 



« 



2.3 Program Offerings 

2.3.1 Adequacy of educational opportunities 

2.3.2 Diversity of program offerings 

2.3.3 Trained Staff 

2.3.4 Modified course presentation (handicapped) 

2.3.5 Support services ^ . 
• * a. day care 

c b. auxiliary aids 
c. remedial 
* d. financial - j 

2.3.6 Program cbiiprehensiveness A 

; c - job, social and employability skills 

Note .; Required by federal legislation.^ 1 - 
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2.4 AttitudinaX fesrie^fe 



2.4.1 Stereotyping of person's ability to ( do work 

2.4.2 Stereotyping in career categories 1 

2.4.3 Prejudice about student's ability to ^earti 

2.4.4 ^Individual staff prejudice concerning certain 

groups of people 

2.4.5 Student self-concepts 

Societal Factors 

3.1 Attitudinal Barriers \ . 

» * 3.1.1 Stereotyping person 1 s ability to do work . 

3.1.2 Personal negative self-image 

3.1.3 Job stereotyping* 

3.1.4 Non-accepting attitudes in. society 

* *• 

3.2 Betjaviorial Barriers * * 

3.2.1 Jofy-modif ication 

3.2.2 Job sharing ^ 

3.2.3 Ability to secure satisfactory employment v 

3.2.4 Role model representation 

3.2.5 Policy statements 

3.2.6 Hiring* practices 

3.3 Economical Barriers * 

• ? \ 

^ 3.3«1 Assure availability of suitable Employment 
3.3.2 Inform completers about available employment 



> v 



* PARTICIPATION 



1.0 Enrollments 



\ 

^ „** 1*1 Disadvantaged 
•, J 1.1*1 • Econom 




/ * 



r 



Economically 
l.l.A Academically * 

1^2 Handicapped 

1.2.1 Mentally retarded 

' i.2.2 Hard of "hearing 

1.2.3 Deaf_ ^ \ 

1.2.4 Speech impaired 

1.2.5 Visually handicapped * 
1^.6 Emotionally disturbed 

1.2.7 t)£thopedlcally« impaired 

1.2.8 Other health impaired 
1.2.9, Specif ic lear ning ^ disabled 



E* 1.3 Minorities ■■■ % x « , 

American ^n^ian/41askan Native 
Asian Amc^foan/PaciSic Islander 
Blacky not Hr^paijlc 
Hispanic 

White, not Hispanic 

/ 

** lv4 Limited English Proficiency 
* 1.4.1 Spanish dialect %. 
XCh.2 Italian dialect ' 
1.4.3 Other 



* 1.-3.2 
1.3.3 
1.3.4 
1.3.5 



** 1«5 " Sex Designation 
1.5.1 Female \ 
I 1.5.2 Male 

'■ 1.6 Age (Elderly) 



** 1.7 Instructional Setting -(handicapped) 

1.7.1 Regular class • ' * 

1.7.2 Mixed class 

t. 7*3 Separate /class 

\.T%% Separate facility * - 

l./;5 Other . 

Note . ** Required by VEDS V , due December 1, 1980 - 
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, PROCESS 

1.0 "Additional Services" "• 

1.1 Administration Related (SEA and LEA^Po^cy Statements) 



-A 



J 1.1.1 Program 

% a. diversity ' • 1, ■ 

b. duration (flexibility) " 'V "\ 

c. class size ■ , . 4 : 

d. . staff /student ratios 

e. migrant reciprocity 

1.1.2 Job Placement *X 

1.1.3 SEA and LEA/Emplx>yer Relationshipfe 

- 1.1.4 Staff ^ i - 

*** a. patterns' (racial /ethnic and female) 

b. « up-grading # 

c. role models / 

1.1.5 Interagency Agreements 
7 j- - a. vocational education/ special education 

b. vocational education/CETA 

■ Cm, vocational education/vocational rehabilitation 

^ 1.1.6 Advisory Committee Utilization 

a. state 

b. local 

c. craft * 
* 1.2 Guidance and Counseling Related 

v * .. 

-1.2.1 Student testing and assessment 

1.2.2 Career informa^n 

1.2.3 Comprehensive cSreer counseling 

t * 1.2.4 Non-discriminatory counseling materials and activities 
* l»T.5 Interpreters for LESA and hearing impaired students A 



Note . * Required by( federal legislation. > ' y 
*** * Required bywEDS, due December 1, 1981.. 



3 Instruction Related * 

*. •* ■. ...'*. 

1.3.1 Facility and equipment availability* 'j. . m > 

* 1.3.2 Facility and equipment modification 

* 1.3*3 Bias free materials (allS special populations) 

1.3.4 Individual (tutor) instructor 

1.3.5 Individualized instruction - 9 

1.3.6 Individual education program (IBP), (handicapped) 

1.3.7 Individual work program \lWP) f (handicapped) * 

1.3.8 Instruction toward 

% a. independent living 

b. personal care 

* 'c* work adjustment 

d. social skills . > 

1.3.9 Recreational and social activities 

1.3.10 Non-bias, non-stereotyping and f non-discrimination 

a. among staff 

b. among non-special* population enrollees J 

* 1.3.11 Cooperative evaluatiOn/work study 

* 1.3.12 Apprentice training 

4 Placement,, Related 

1.4.1 Employer consultations • 

1.4.2 Job matching (handicapped) 
1.4.3 > Job res true ting (handicapped) 

1.4.4 Job follow- through x ' 

1.4.5 Recommendations for enrollees lacking or haying poor 
work histories 

* 1.4.6 Employer or prospective employer discrimination 

" * a. recruitment 

* b. hiring 

* c. placement 

* d. assignment to work task 

* e. hours of employment 

* levels of responsibility 
g» pay 



Required by federal legislation. 



OUTCOMES \ 

1.0 Student Achievement . ' v •— ^— ' 

• ..- . -; . s • : , 

* 1.1 . Knbwledge and Skills for Successful Employment 

1.1.1 basic educational 

1.1.2 technical ■* " - 
' 1.1.3 employability 

1.1.4 work adjustment * ^ 

1. 1.5 personal * - - - 
1 • 1 • 6 social s\ 

1« 1.7 independent living 

1.2 Acquisitions' in TXhe Affective Domain , ' ' * 

(i.e., interests , Attitude s , appreciations ,'values, and 
emotional $ets or biases.) 

2.0 Successful Program Completion 

2.1 Grades 11 and 12 — - 

2.1.1 limited competency acquisition to meet individual needs 

2.1.2 program competency acquisition « 

2.2 Post secondary . 

a 2.2.1 limited competency acquis! tipn to m$et individual needs 

" 2.2.2 program competency acquisition * > - 

* * 

2.3 Adult , . ^ 

2.4 Apprenticeship -a * 

3.0 Student Placement (completers/leavers-racial/ethnic/sex i 

designation/disadvantaged/limited English .and .< 
handicapped) 1 1 * ^ 

;***-3»l Employed (rate) * 

*** 3.1.1 in field related to training t 
"\ . a. civilian ' ■ > \ . 

b. military 

*** 3.1.2 in field riot related to training \ 
a. civilian ^ 
> b* military . 

*** 3.1.3 salary rates \ J 
/" *** 3,1,4 employer satisfaction *-\ / ♦ ^ 

3.1.5 other 



Note . * Required by f ederal x legislation. 

^*** Required by VEDS, due December 1, 1981. 



\ 



4 



4.3 Salary Increases 

4.4 Reactions to Training 



Note. .* Required by federal legislation; 

***• Required by VEDS, due December 1, 198U 



*** 3^.2 Unemployed (rate) ' 

*** 3.2.1 seeking employment and not pursuing additional 
* education 

*** 3.2.2 not in labor force and not pursuing, additional % 
education >■ \ \ / / ^ 

;■•;/■*** 3.3 Pur^ijing Additional Education _ ■, ; * - 

,*** 3.4 Status Unknpwn 

' .>..••* . ■ ■ , ■ 

.... , .. ( ..*,•*■•■'*■■■■ • ■ ' • 

4.0 Successful Employment Over Time 

, 4.1 Duration > v V. ' 

° 4.2 Promotions • 
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A Causal Model Showing the Relationships 
of Access, participation ^ Process, and Outcomes 

• • ' : ■ ' ' • . ■ ' '" t " ' ' : ; 

As shown in the matrix (Figure 3), participation is a major 
indicator of access while outcomes are major indicators of 
process. This does not mean/ however, that participation is a 
sole function of access factors and outcomes are sole functions 
of process*. The model (see* Figure 4) illustrates the causal - 
relationships of access factors^ participation, process, and 
outcomes. 



Vocational education takes place in a relative task 
environment. Several factors in the environment influence the 
delivery of vocational education. Figure 4 illustrates the 
cyclic and sequential relationship of variables as they 
contribute to the delivery of vocational education in the task 
environment. Endogenous "factors in-.. the model are variables * 
that are found in the relevant task environment while exogenous 
factors are variables, such as certain pressure group's, which 
emanate from outside the task environment, 

It is interesting to note tnat within the task environment 
all the major variables under consideration have at leagt one 
arrow from another variable directed to them. This illustrates 
the complex nature of the relationships that exist among the 
variables and the compound inc^effects of o.ne variable on other 
variables. Other unaccounted for intervening variables 
(represented by straight arpVws) affect the variables under 
consideration. 1 



In Figure 4, vocational education delivery can be viewed as 
a process that can result in both short term outcpmes and long 
term outcomes. A key to the delivery of vocational education 
is the process component which deals with manipulation of the 
Environment <and its contributing factors and elements. 



Access 

Access refers to those factors which directly influence the 
number and kinds of special population enrollment; facility 
factors, educational factors, and societal factors. Figure 4 
shows that access factors are also influenced by the results of 
student's outcomes. For example,, some modifications in the : 
admission criteria may occur as a, result of ^tuderfts 1 outcomes. 
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A causal diagram showing the relationships of access, participation, 
process, and outcomes. . Y 

, J 42 - • - 



Facility factors , These factors have application chiefly 
for the physically handicapped. As shown in the outline on 
access, all of the raafnagted subf actors must be ^evaluated to see 
that they conform to federal specifications. j** 
Without these special considerations the physically handicapped 
would not be able to participate in vocational education ; 
programs, and would be deprived oc habilitative education. 

• -* • 

Educational factors . The Access outline show that there 
are four subfactorg to be considered in the evaluation of 
programs for special populations: recruitment, admission 
criteria, program offerings, and attitudinal barriers. Under 
recruitment, both public notification and promotional efforts 
are required by law to be nondiscriminatory. In addition, 
admission criteria must be nondiscriminatory. 

Support services can provide access to those who might not 
otherwise find it possible to attend vocational education 
programs, federally funded agencies are required to determine 
whether existing services (e.g., day care, financial* 
assistance) are provided in a nondiscriminatory manner. 

Vocational education has the responsibilty to make program 
offerings as diverse and comprehensive as possible so that 
participants can follow their talents and motivations into 
fields of personal interest. In addition to teaching 
job skills, programs need to teach students employablity skills 
(how to get a job) and the social skills they need to cope with 
the job situation. 

Attitudinal barriers are possibly the greatest deterrent to 
entry into occupational education and subsequent employment. 
Preconceived ideas abouf ability to perform may llad to placing 
special population individuals into training for only lesser 
job tasks. If negative self-concepts are held by students, 
motivation to learn may be lacking, and little learning may 
takes place. 

Societal factors . Schools can have a part in influencing 
people in society to develop accepting attitudes and behaviors^. 
Behaviors in society influence a person's occupational choice. 
Some examples of behavioral barriers which may prevent special 
populations from enrolling in vocational programs are (1) when 
job modification and job sharing are not iftade available to the 
handicapped, (2) .when only the lowest skilled and lowest paying 





» - . ■ • ' ^ . 

job& are made available to minority individuals and women , aria 
(3) when hiring practices and/or policy statements discriminate 
against any of the special population groups. 



Participation 

J Participation % refers to the number and kinds of special 
populations enrolled in vocational programs. As seen irr 
Figure *4, this is principally influenced by two factors—access 
an'd* student outcomes. The type of facility directly influences 
the type and kind of special population enrollment. Further^ 
the success of special populations, both in school and in job 
situations also has an influence on t whether or not these 
individuals^ attempt to eftroll in vocational programs. f 

' Information about participation is obtained by collecting 
accurate enrollment data by subgroups in USOE two-*digitf i< and 
six-digit code designations for each of the vocational 
education service areas and specific vocational education r* . 
programs. Reference tto the Participation outline reveals that 
data on those special population groups which are preceded b# 
two asterisks are required to be collected by states duting 
Year I (1979-1980) of the Vocational Education Data System 
(VEDS) while those groups preceded by three asterisks are 
required by VEDS reporting system for Year II (1980-81). 
Designation as to instructional setting for the handicapped is 
required by VilDS effective Year II. 



Process 

Process refers to the educational environment which is 
manipulated or modified in some way so that learning can take 
place. Vocational education students in general are taug ht i>Q 
a modified environment and learn occupations as a resu^fr But: 
when special populations are subjected to vocational education, 
special process considerations need to be provided. These 
special process considerations are termed "additional 
services" as found in the Education Amendments of 1976. 
Additional services that are needed by any one individual 
student in a vocational education program will vary greatly 
because of the diversity and unique needs found among special 
population groups. m 




Additional services have been divided into four factors: 
administrative related , guidance and counseling related, 
instruction related, and placement related. Figure 4 shows 
that process is influenced by two important factors—numbers 
an<} kinds of special population enr6llnfehts (participation) and 
student behavioral changes. 

Administrative related . One of the factors included in 
process is administration. Administration is heavily 
influenced by forces from both inside , and outside ^the , task 
environment. Administrative program policy, at the* state and 
local levels must afford special populations the opportunity to 
succeed in learning an occupation. A diversity of programs may 
need to be made available, time to complete a program may need 
to be altered, class size may need to be changed to alloy for 
individualized or small group instruction (which would ydecrease 
fcfce teacher/student ratio), and reciprocity of credits earned 
Stx one vocational education environment may need to be accepted 
by the institution receiving a transfer student. At the same 
time administration needs to be concerned about appropriate 
nondiscriminatory job placement for special population* 
students. . 

Administrative staff also need to reflect representation 
of women and persons representing differing ethnic groups. The 
VEDS reporting, system requires staffing pattern data for Year 
II. At the same time, employment \±n administrative positions 
must allow, for advancement or up-grading and present positive j 
role models to those who may otherwise be too discouraged to / 
seek administrative positions in vocational education.^ 

Guidance and counseling related . Evaluation -must examine 
the materials used and the activities performed by counselors 
to assure that neither materials 1ior activities are 
discriminatory. Evalua tors, also need to ensure that ^ 
interpreters *are provided for language- and hearing-impaired 
students. The Education Amendments, of 1976, in general, 
specify that guidance and counseling may include evaluation of 
the' kinds and frequency of student testing and assessment, th^ 
amount of diversity of career information, and the 
comprehensiveness of the counseling. * * 

Instruction related . As seen under Access, facility or 
equipment should not constitute barriers to vocational 
education programs for special populations. Therefore, for 



certain learners^ facility and equipment modifications may need 
to be ma#fe especially in cases where learners are handicapped* 
In addition, instructional materials u3ed must" be free from 
bias if published after the civil rights legislation of 1972. 

Cooperative education, work-study, and apprentice training 
are additional means for learning an occupation. If any of 
these programs are available in a school system, provisions 
must be made to prevent discrimination against special V 
populations who may wish to participate. 

At times, there may be ,a general need to provide tutorial 
or individual izq£ instruction to special population groups. 
This kind of instruction may be generally required in* the case 
of handicapped students who have mental or .emotional 
disabilities. P.L. 94-142 requires that all handicapped 
secondary Vocational education students have an Individualized 
Education Program (IEP) which should coincide #ith an 
Individual Work Program (IWP). 

Special educators have the responsibilty of teaching basic 
skills to disabled students to enable them to become \ 
independent in their living. They must function within a 
community with as little assistance as possible. Handicapped 
students need to be taught personal care 'and social skills. 
All Qf these need to be reinforced within the occupational 
training program provided by vocational education, especially 
for those handicapped who are mentally disabled in some 
capacity. Recreational and social activities within the 
institutional environment are as important to special 
populations as ,they are to regular students. Evaluators need 
to check to see that discrimination does not occur and that 
these activities include, to the extent possible, all students. 

Placement related . Another area to which evaluators should 
pay particular attention is employer or prospective employer 
discrimination in recruitment, hiring, placement, assignment to 
work tasks, hours of employment, levels of responsibility , and 
pay. Evaluation of practices in these areas is mandatory. 
Other elements evalaators may choose to examine are job 
matching and restructuring especially for the handicapped. Job 
follow-through for al^. special population group^is important. 
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Outcomes 1 

PlannedJor expected outcomes of vocational education 
programs include student achievement (student behavioral 
changes) , successful employment completion, student placement, 
and successful employment over a period of time. The latter is 
not a legislated requirement, but it is of important in 
evaluation. The model .(Figure 4) shows that student outcomes 
are influenced by process, participation, and actfess as well as 
outside factors^ 

" Student achievements In a vocational education program one 
of the outcomes is the acquisition of knowledge (a cognitive 
function) and occupational skills (which include cognitive, 
psychomotor, and affective f unctionings) • It is expected that 
students will achieve in, basic educational knowledge, technical 
knowledge, occupational skills, employability skills (how to 
get a job), and skills required to hold a job (included as part 
of work adjustment). Requisites for community living dictate 
that individuals acquire independent living skills and be 
mobile and functional in the community. In addition, they need 
to develop personal and social skills that make them acceptable 
in the community, particularly in their work setting**- 
Interests, attitudes, and values are elements of the affective' 
domain that are influenced by vocational programs. 

. * ■ ^ - . - ' ■ 

Successful program completion . While successful program 
completion is perhaps the goal which most vocational educators 
hope to achieve for their students, this dods not always occur. 
One student may meet all the demands of program competency 
acquisition while another student may only acquire limited 
competency which however, satisfies the individual needs of 
that student. It seems important to know, for example, that a 
mentally handicapped student ;has achieved "X" number of 
competencies even though that student may not be able to 
achieve the Entire l^st of competencies required for completion 
of. an entire program. Under the Rules and Regulations for 
Section 504 of the Rehabilitation Act of 1973, (Section 84.4 
(b) (2)) the law requires the following: 

* ' . 

For purposes of this part, aids, benefits, and services, 
to be equally effective, are not required to prod ice the 
identical results or level of achievement for 
handicapped and non*handicapped persons, but must afford 
handicapped persons equal opportunity to obtain the same y 
results, to gain the same benefit, or to reach the same 
level of achievement, in the most integrated settirig / 
appropriate to the person's needs. - 
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Student placement * Youth unemployment xs a major ^concern 
to the Congress of the United States. Unemployment levels of 
young adults, especially young adults belonging to minority 
groups , helve, increased over the last j#ecade. Vocational 
education is being held accountable for placement of its 
students in a stricter sense -than has been the case, in the - 
pas£. As can be seen by referring to thewnatrix outline on 
Outcomes, the VEDS reporting system for Year II is requiring 
placement information on completers and leavers of vocational 
education ^programs whether they are regular students or 
students rtom special population groups. Placement for some 
special population individuals may be some thirw other than job 
placement. For some handicapped individuals, ©or instance, 
such things as sheltered workshops or additional training may 
- be more acceptable. [ 

SjK5tfessf ul* employment over a^period of time . The rates of 
vocational program completers and leavers whet are successfully 
-employed ov^r a period of time should provide' vocational 
^^jiication important information by which to determine the 
Wfi^0Gtiveness pB programs. Also important is thte information 
Concerning the factors which contribute to successful long-teirm 
eroplopien t . Such information will be of great value in 
determining the kinds of program improvement needed to increase 
program effectiveness. * 

The Self-Assessment Instrument 
* ~ : ! — ' ~~ 

The self masses smen t instrument was designed to help states 
assess the adequacy of their vocational education systems to 
evaluate programs and services for special populations, 
especially Meeting both legislative and planning requirements. 
The development o^ the instrument involved the following steps: 
•■ * ' Vtf$ - . . . 

1. An initial instrument was derived by the project staff 
from the elements of the evaluation matrix* This was 
circulated among the staff of the Evaluation and 
Policy Division. A meeting was held at which 
suggestions and criticisms were given and noted. 
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2. The instrument was reviewed and tested by the four . 
cooperating states in rating the states' 
respective abilities to evaluate programs and/ 

% services fo,r special populations enrolled in 
vocational education. States 1 reactions with 
regard to the validity of the criteria and 
measurement indicators being used were noted. 

3. A revised instrument was then constructed* This was sent 
to. selected staff members of the National Center and 
state vocational departments for review. Suggestions and 
criticisms wqre noted. 

4. A final instrument which states may use to rate their 
abilities to evaluate programs and services for 
special populations in vocational education was then ^ 
constructed (see Appendix A). 



Ranking of Level of Difficulty 
of Problems in Evaluating Programs and Services 

: . f oy Special Populations ~ v 

Level of Difficulty 

During the year the technic^al^evaluation assistance team 
worked with states on evaluation for special populations a 
number of problems were identified. A number of state people 
working in evaluation in several states were asked to provide , 
their perceptions of the level of difficulty or these Emblems 
This ranking was done informally on an "individual baj^fover a 
period of several months. Each person was asked to rate the 
degree of difficulty it posed to that individual's agency. 
Some twenty experienced evaluators participated in this * 
exercise. 

Following is the order of ranking with the most difficult 
listed first and the least Sifficult* listed last. 



Problem . 



Conducting longitudinal follow-up of './ 
special populations • „ 

Measuring student achievement ~ ■■ i 

of special populations V < 

Securing adequate financial support to 
perform evaluation of' programs and 
services, for special populations & 



Securing organizational support for the 
evaluation of programs and services for 
fecial populations 

Securing personnel to perform evaluation 
of programs apd services for special 
^populations 

% " • . , 

Determining criteria for the evaluation 
of prpgrams and services for special 
populations 

Determining criteria for the evaluation 
of programs and services for special 
populations - 

Using special population evaluation, 
information / 

Constructing/developing evaluation 
instruments for evaluation of programs r 
and services for special populations 

Interpreting results of evaluation data 
about special populations 



Rank * .Problem '* Mean 

-10 Understanding the requirements of : *. 

pertinent legislation regarding the J > 
evaluation of special populations 

12 Determining special population enrollment 2.84 

according to the VEDS classification at 
the USOE six-digit code 



13 Performing process evaluation of special jZ.79 
• population^ . % 

14 Preparing reports regarding evaluation of 2.78 
programs arid services for special 

) populations , 

15 * Preparing a well-defined and systematic 2.75 

plan regarding evaluation of programs and ' 
services for special populations enrolled 
- in vocatifoflal education 

16 Determining special population enrollment 2.64 
% according to the VEDS classification at 

. ,' the USOE two-digit code , *~ 

17* Determining program accessibility 2.58 

( - \* > 

17 t ' Conducting student follow-up. of special 2.58 
populations 



19 1 Analyzing results of evaluation data about 2.50 

> special populations 
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Notes 



V ' 



2. 



Federal Register, Vol. 42, No. 6 "(May 4, 1977) , p. 22679. 

Selected elements of the evaluation* matrix were included 
the content of the self-evaluation instrument. Elements 
which are not generally the responsibility of state 
evaluation units were purposely excluded. 
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, CHAPTER IV * ' 

QISTINCTIVEf, FEATURES OP EACH* STATE 
VOCATIONAL EVALUATION SYSTEM 



ft- 



/ This chapter presents a brief summary of the distinctive 
features of each cooperating state's vocational education evaluation 
s^stem^ discussion includes both the secondary and pos tsecondary 
systems except in Alabama where- the bostsecondary evaluatiop^system 
is still under development and is no|w being pilot tested • 4 ^^ 



Alabama 

Under the Division of Vocational jEducation Services the 
leadership role of evaluating vocational programs in the state lies 
with the Research Coordinating Unit (RCU) in coordination with the 
Management ; Information Service (MIS) ^nd the Prdgram Supervision 
Unit (PSU), as'^shown in Figure 5. 



Management Information Service (MIS) 

* ■ _ 

The MIS compiles and publishes, enrollment and follow-up data* 
The MIS supplies standard forms arid standard instructions to/ local 
program administrators for report ing^hrollment data arid conducting 
•an annual follow-up study and fot reporting the results to .state 
staffs The state furnishes local vocational directors and staff 
with labor" market demand and supply data and follow-up data tor use 
in preparing/ their annual program applications • The MIS also 
furnishes the district vocational specialists with datfa collected to 
help them in their review of local vocational programs, 



Program Supervision Unit (PSu1 



4? 



Th§ district vocational specialists t under the PSU conduct the 
first phase of the program review by making an analysis of each * • 
vocational program in the local School being studied. Thisf is 
preceded by a self-evaluation by each teacher and followed by an p 



on-sxte review team. 



spec 



The program review iftstrument completed by the district 
ialist is made available/ to the appropriate review team 



members 



Budget 
Coordination 



Support 
Data (MIS) 

— TT 



Collects selected 
data by mail from 1 
teachers, students, 
former students and 
employers 




Research 

Coordinating Unit 



% (RCU> 

A member of RCU serves as review 
team leader \ t 
Develops review instruments 
Prints reports; 



Program 
Supervision 



Phase 1 On-site Visitation 
Supervisors review 
and visit each , 
instructional program 
in the system; ^ 



\ 



Review 
Team 



Phase 2 On-site Visitation 
A team of state and local 
staff visits the system 
to evaluate the total 
vocational program 
within the system. 



Perform 
Self-evaluation 



Figure 5. Organizational chart of. the Alabama Division of Vocational Education 
indicatina secondary ^evaluation functions. 

• • . • - ' -b • . . • - . 



prior to the team interviews. The review team. members study the 
completed instrument and plan the interviews. In addition ,. the ■' - 
summary sheets completed by' the district specialists are reviewed by 
the team leader. 

Research Coordinating Unit (RCU) - 

The RCU Hakes the leadership role in the total vocational 
education evaluation system. It develops evaluation instruments for 
use by district vocational education specialists and the on-site 
review team. It assumes a leading role in the on-site team review. 

The review team consists of state department members, local 
staff from other systems , and sometimes teacher educators and 
employers. An average of ten members compose a team, and a member 
of the RCU serves as review team leader. The main purpose of this 
review is to evaluate the total vocational program within a local 
school system. * 

' V 

Basically, the following procedures are followed in conducting 
on-site team review visits: 

1. The team leader consults with district vocational education 
spegialists regarding the results of program review. He/she 
collects and reviews pertinent information relative to the 
specialist visit and of importance to the work of the team 

* reviews. * 

2. Prior to the actual on-site review, the team leader makes a 
planning visit to orient local school staff members to the 
objectives and procedures of the on-site review. He/she 
also collects selected information necessary for planning 
the interview schedule and work of the*review team. 

3. Immediately preceding the on-site review, the team leader 
conducts a team orientation which includes a reyiew of the 
pre-visit information, a summary of the district 
specialist's reviews, and interview scheduling. 

m 

4. The on-site review consists primarily of interviews 
conducted over a two-day period. 

5. Using a prepared interview schedule, review team members 
interview local administrators, counselors ,* vocational 
teachers, vocational and non-vocational students, former 
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students, board members, advisory council members, and 
employers* In larger systems, only a sample of teachers 
may be interviewed* * 

6. A report, regarding the findings o^ the visit, is prepared 
by the team members. The report contains findings, 
recommendations, and suggested activities* At the end of 
the teview, the teajn leader gives an oral preliminary report 
to th£ local superintendent and selected staff* 

7. The report is written, printed, and* distributed to local 
superintendent, who receives multiple copies, and to 
selected state staff. 

8<» Local staff review the report and develop tentative 
follow-up plans in response to tfie recommendations. 

9* Appropriate state staff assist the local staff in developing 
the tentative follow-up plan. 

10. The state vocational director and * the team leader meet with 
the local superintendent and other local staff to finalize 
the f pi low-up plan. 

11. The' district specialist and appropriate state staff provide 
technical assistance to local staff in implementing the 

♦ follow-up plan and they monitor progress. 

California 

In California, there are two delivery systems for vocational 
education. At the secondary level, delivery of vocational education 
lies with the State Department of Education, specifically, the 
Vocational Education Division* At the postsecondary level, it lies^* 
with the California Community Colleges 1 System under £he Office of 
the Chancellor. In view of the foregoing organizational 
arrangement, there are two systems for the evaluation of vocational 
education. ^ 

■ ^ ' 

Secondary Vocational Education Evaluation 

Responsibility for developing, initialing, and coordinating the 
secondary vocational education Evaluation system in California lies 
with the research aj)d evaluation consultant in the Support Services 
Vocational Education Unit. This individual, with the involvement 
and assistance of a number of people in the department and in close 
coordination with postsecondary evaluation, established the system 
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unit indicating the secondary education evaluation 
functions. 



described in this section. The personnel who conduct the ^sample 
verification visit of th^ Program Assessment - Vocational Education 
(PAVE) are furnished by the Field Operation Section of the Office of 
Vocational .Education. This section, which has three regional 
off ices, provides technical assistance to districts in developing 
plans, collecting data, improving program management and 
accountability, making statistical reports, and performing 
evaluation. The Field Operations Section works cooperatively with 
the Office of Program Evaluation Research of the State Department of 
Education (see Figure 6). 

California's system for evaluating vocational education at the 
secondary level includes four ma jc^r activities: (I) the Program 
Administrative Reviews ( PAR) , which provide for documenting, 
efficient administrative practices; (2) the Program Assessment 
Vocational Education (PAVE), which is an evaluation of instructional 
program effectivenss; (3) the Desk Review and Field Audit (DRA), 
which reviews accounting procedure; (4) and the Follow-up of > 
Students and Employers (FUSE). 

To accomplish the accountability plan, California divided its 
districts and regional programs into four groups. Each group was 
designed to be representative of the public agencies which deliver 
vocational education services in thte state. It was thought that by 
gathering administrative, vocational instruction programs, — 
accounting, or student follow-up in forma tfitbrv from any one of the 
four sample groups in any , one year, accurate estimates can be made 
about how all of the districts and regional occupational centers 'and 
programs (ROC/ Ps) stand on all areas, of inquiry. 

Program Administrative Review (PAR) . PAR was designed to insure 
that each administrative unit — the schoo^^district or ^gional. 
occupational program administration — takesT 1 those measures necessary j 
for the proper administration of vocational education funds . PAR i^fi 
applied at the central administrative level, that is , ^tgte^^ool 
district or regional occupational program central administrative 
office.* PAR attempts to help vocational education coordinators and 
other administrators understand the laws and regulations under which 
they operate. 




J 



PAR serves the dual purpose of informing instructional service 
agencies of their administrative obligations and of providing a 
means for helping 1;hem to devise methods for supplying the 
documentation and records nefcessary to satisfy audit requirements. 

Eacn^chopl district andjregional occupational unit in the PAR 
sample is visited by a statqf consultant. Using a comprehensive 
interview and observation schedule , the consultant determines how 
well the local agency is meeting the requirements for Record keeping 
and fiscal administration. Ways of improving district 
administrative practices are suggested, and a written report of 
recommendations is made to each agency. Subsequent follow-up is * 
done to determine th£ courses of action that agencies have adopted ' 
in response to recommendations and their effectiveness. A final 
letter of agency status regarding PAR is sent after the follow-up. 

Program Assessment - Vocational Education (PAVE) . PAVE 
describes and determines the effectiveness of the 'educational 
process for each instructional grpgram. Each year one of the four 
groups is involved in this process. 

The first part consists of obtaining evaluation information from 
each program. Each program in each school or regional occupational 
unit responds to a questionnaire. The questionnaires are sent 
through the regular school or regional program distribution system. 
Those completing the questionnaires are requested to consult and/or 
solicit the help of teachers who provide the program , instruction. 
Questionnaires are assembled by each district and regional 
occupational unit and mailed to the vocational education unit. A 
quea^ionna i re is required from a specific program once in a 
four^year period. 

The second part of PA VE^cSns i sjMupf ^ visit by program area 
specialists in agriculture,' of fie* ■Lstributive, consumer and 
hojnemaking, occupational related W making, industrial, health, 
industrial arts, and work expetlenCT^education to a sample of 
questionnaire respondjjifes . These specialists verify and amplify the 
program reports by -observing facilities and services and by 
questioning students, teachers, counselors, and administrators. 



Area specialist visits are. designed to cause the least 
disruption possible to instructional programs, but it is necessary 
to talk to individual instructors and counselors and to obtain brief 
questionnaire responses from students. 

The program area specialists summarize their observations in 
oral and written reports which are delivere'd to the school or 
regional unit to help the selected local personnel in making 
instructional program review. 
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Desk Audits and Field Review (DRA) 

DRA requires that all participating administrative units sul^ait 
selected financial reports to 'the- State Department of Education .for 
review. These records include analytical statements of vocational 
education program expenditures for the disadvantaged/ the • 
handicapped , and other target groups, 

';. In the event that diff iculties' are encountered in understanding 
the records, a state agency representative calls oh the district or 
regional program off ice and completes the work on-site. 

In addition, DRA includes an on-sit^e review of the financial 
records o£ a small sample of public educational ' institutions 
receiving financial assistance under Public Law °9 4-4 8 2 and Public 
Law 94-40. These sample reviews are performed by an independent 
outside contract agency. 

Desk reviews and field audit activities are scheduled in ' 
advance. Institutions involved are informed of their involvement 
ahead of time so that they can assemble "and present required 
records. - — * " 



When record keeping discrepancies or irregularities are found, 
effort is made to confer with the instigation on ways to correct or 
overcome the problems. 

> ' . 

Follow-up of Students and Employers (FUSE) . FUSE conducts a 
questionnaire survey of former vocational education studepts 
(completers and leavers) which provides information for phe 
participating district or regional occupational program. This 
information helps them justify and improve 'their programs and 
furnishes data for annual reports to the national Vocational 
Education Data System. * j 

Prior to completing vocational education programs, students are* 
informed by their \ teachers about the follow-up study a^id encouraged 
to respfond to questionnaires that might be sent out from their 
schools. # > K 
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In the year that they participate in FUSE, districts and 
regional* programs prepare rosters of previous year participants L 
which contain the necessary address and telephone locator 
information for sending questionnaires. The roster matches the 
previous year's statistical report of student completers and. leavers 
by program (Vocational Education 48 for district* an<3 the equivalent 
for regional occupational a programs ) . v . 

The materials needed to conduct the survey (questionnaires 
and master cover letters) are prepared by the state and forwarded to 
the participating units. The units conduct two mailings two weeks 
apart. > . 

The district or regional occupational program responsibility is r 
cdmpleted by conducting a brief telephone survey of ^a sample of 
those not responding to the mailed questionnaires.. 

Questionnaires and telephone survey forms are analyzed by the 
state, and a report is sent to each participating district. 

Districts help conduct surveys of those employers identified in 
the student questionnaires. Questionnaires are mailed and a 
follow-up to those who have not resjJond^d within a two-week -period 
is mailed out. Results are analyzed by the state and a report sent 
to each participating state. 



Postsecondary Vocational Education Evaluation 

At the postsecondary level, the leadership role for conducting 
vocational evaluation lies with the .Office of Program Evaluation and 
m Approves. Through its evaluation specialist, it provides technical 
assistjpce in evaluation to community colleges. The office also 
works cooperatively 7 with the Office of College Services especially 
in terms of providing assistance in implementing recommendations. 
Additionally, the evaluation activities are conducted in conjunction 
with the Chancellor's. Of f ice Information System (see^Figure 7). 

Basically, two methods are employed at ^community colleges to 
evaluate vocational education programs. These pre the Community 
College Occupational Programs Evaluation System (COPES) and the 
Student Accounting Model (SAM). Additionally, the Program 
Administration Review (PAR) instrumentation developed by the 
^secondary system is used to address compliance of college programs. 
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Figure 7, Organizational chart of the California community 
* colleges indicating^fche postsecondary vocational 
education evaluation functions 



Community College Occupationar Program Evaluation System 
(JCOPES) , COPES was established in l97l/as a cooperative undertaking 
of community college leaders and the Chancellor's Office of the 
California Community Colleges, Since then the system has been 
applied at a majority of all of the community colleges of the state. 
The basic goal of COPES is to improve the quality and availability 
of occupational education in California community colleges. The 
system has been modified to reflect the quantitative requirements of 
Sec, fl2 of the vocational education'legislation of 1976, Plans for 
the validation process of COPES have been severely curtailed due to 
fiscal limitations imposed as a 'result of Proposition 13/ 

Student Accounting Model (SAM) , SAM is a system of procedures 
(or model) constructed for the purpose of improving occupational 
student follow-up in California community colleges. The model has 
been sponsored by the Chancellor's Office and developed by a ■ ^ x ~ 
consortium of twelve leaders in California Community Colleges, A„ 
fundamental component of the model is a uniform method for 
classifying courses and identif iying occupational majors so that 
non-continuing students can be categorized for various approaches to 
follow-up. 



Colorado 

The major responsibility of conducting a systematic and 
effective vocational education evaluation for both secondary arid v- 
postsecondary programs lies with the Occupational Education Division 
" of the Colorado Stat£ Board for Community Colleges and Occupational 
^^Educa^ion. Within the division, the following branches are involved 
in the process: Planning and Support Services Brdnch and Program 
Services Branch (see Figure 8), * v 



Planning and Support Services Branch (PSSB) 

The PSSB performs two basic functions in the total vocational 
evaluation system, First, thef branch develops and compiles an 
annual accountability report which addresses, among other things, 
legislative requirements. Second, through its MIS, it meets 
requests of local "and vocational directors and staff , state staff , 
and legislators with approximately 100 standard reports and X, 200, 
special reports each year, Bart of^ its standard reports include 
enrollment and placement data. The" system also prodyces "special" 
statistics like effectiveness ratios and efficiency ratios, .A 
committee of secondary and postsecondary local vocational directors 
and staff regularly review data collection by the system and call 
for any necessary modifications. 
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riaure 8. Organizational chart pf the Colorado community 
" colleges and occupational education indicating 
« vocational education evaluation functions. 
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Program" Services Branch (PSB) . 

The PSB handles .the' supervisory visits and the Comprehensive 
Program Review . Specialist area supervisors receive data reports . 
from t*he MIS to help them in reviewing the local vocational programs 
under their supervision. Using prepared checklists, supervisors 
review programs in their "service area where* CFRs: are not being 
conducted. ^ The area Supervisors also conduct cross-site analyses of 
the data ttiey collect during their site visits in order to identify 
common problems and strengths among the local vocational education 
programs. Fittrther ,\ the supervisors play a major role in the CPRs by 
serving ^s visiting team leaders or coordinators of several visiting 
teams. \ 

Colorado performs a comprehensive program review of five to v 
seven schools per year covering about 150 vocational programs and 
related services. . r 

The Comprehensive Program Review (C PR) consists of three phases. 
Phase I is preparation for evaluation,; Phase II is on-site 
visitation; while Phase MI is follow-up procedures. Basically, it 
employs two evaluative methods: process method and product method. 



Connecticut 

The state of Connecticut does not directly evaluate the schools 
and" institutions that offer vocational education. Instead, a state 
initiated self-evaluation exists for local agencies t^hat receive 
federal funding. The Division of Vocational Education, through the 
Bureau of Vocational Services, reviews 20 percent of those 
evaluations annually. The state-operated/regional \ 
vocational-technical schools are evaluated by an independent . 
afccrediting association-^-the New England Association of Schools and 
Colleges. At the same time, the division also maintains a 
management information system through its Bureau 'of Vocational 
Program Planning and Development (as shown in Figure 9). 

Vocational Program Planning and Development (VPPD) 

The bureau collects and processes statistical program data for 
state and federal statistical reports and f5r program planning. A 
follow-.up study of all vocational graduates is conducted annually 
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and a report of tjhe results is published. The V staff also assists in 
the development of research and exemplary program proposals and in 
the evaluation of such programs. A comprehensive management 
information system (CVEIS) is also operated by the bureau for the N 
Division of Vocational Education. 

Bureau of Vocational Services 

The bureau distributes a "Manual f or v Self-Evaluation of 
Vocational Education Programs" to all local education agencie^. 
Twenty percent of the self -evaluated LEAs are reviewed annually 
following this procedure: - • 

1. Appropriate local, state, and State Advisory Council for 
Voc^axiorial Education of ficials cooperatively evaluate 
programs through on-site visitations. 

2. Evaluation reports containing recommendations are made' by 
the group and are provided to local education agency (LEA) 
administrators, State Advisory Council, and the chairperson 
of the local advisory council. 

3. Each LEA, agency, and institution is requested to provide in 
its annual plan information pertaining to the use of ■ ' 
evaluation results. 

4. At the end of the year, a follow-up instrument is 
distributed to those LEAs/agencies whose programs wer;e 
evaluated the previous ^school year. The purpose is to 
determine if previous recommendations are bei^lg implemented. 

'p 
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The purpose of the National Center Technical Evaluation 
Assistance Project was to assist states in identifying and solving 
problems associated with their evaluation systems. Year I of the 
project involved California, Alabama, Colorado, and Maine* Several 
visits were made to these states in order that project staff could 
observe the evaluation system and interview state staff, members of 
state advisory councils, local administrators, and vocational 
teachers. Project staff also collected relevant documents which 
were analyzed in an attempt to determine the effectiveness of the 
evaluation system. 

On the basis of the above observation and document reviews each 
state's evaluation system was analyzed. Each state analysis 
contained a brifef explanation of the problem, several alternative 
solutions, and the advantages and disadvantages of each solution as 
viewed by the project staff. States were encouraged to use these 
alternatives as a starting point to begin mapping strategies for 
improving their evaluation .efforts. 

Project staff met with coordinators of evaluation and vocational 
administrators to determine the technical assistance which would be 
most appropriate in the improvement process * Approximately nine 
months 'later project staff assessed the extent to which-^ach state 
had initiated actions designed to solve the problems identified in 
the Year I final report. This is a report of the states 1 efforts to 
strengthen the weaknesses seen in their evaluation systems. 

Alabama 

Alabama used the school year of 1977-78 to develop and field 
test an evaluation system for vocational education. The staff af / 
the Research Coordinating Unit were mainly responsible for this^" 
work. In May of 1978, the technical assistance project staff had an 
opportunity to observe onfe-<5r the field test program evaluations in 
an area vocational school and to be involved in analyzing what had 
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been learned in the Jtest and in discussing solutions for apparent 
weaknesses. Following ^re the major problems^identif ied and the . * 
solutions adopted* y 



Problem 1 :\ How to increase the involvement of a variety of people 
V"v in evaluation of programs. 

Explanation ; At the time this problem was identified, it was / > 
state policy that only selected state staff and local vocational ) 
administrators could serve on evaluation teams. Teams were 
observing programs and interviewing teachers, students, 
administrators, and employers. While on the surface thjLs might 
appear to be faitly extensive representation, the final 'decis-ion as 
to what was written into the evaluation report depended entirely on 
individuals strongly influenced by feelings toward perpetuation of 
the present system. This does not question the dedication of those 
individuals to improvement, but it does indicate the potential for a 
sinrgleness of viewpoint, A panel, with more diverse backgrounds 
could be-expected to view needed changes from different perspectives 
and at least expand the list of possible alternative solutions. 
Another consideration in choosing individuals to serve on teams 
would beoto select individuals representative of the special .needs 
populations participating in programs or needing braining. This 
would provide a view of programs and needs which might be quite 
different from the commonly expressed opinions. 

Results : The evaluation systems as presently constituted shows 

greatly expanded involvement of many diverse individuals. The 

original policy was changed to allow representation by many 

individuals having ^interest in vocational education, 
* • 

Presently team doing, program reviews may be made up ol state 
staff, teachetf educators, graduate students in teacher eduction, 
local administrators and teachers—both vocational and 
non-vocational— special education staff, employers, counselprs, and 
members of the State Advisory Council for Vocational Education , 

Problem 2 : Follow-through oa recommendations. 

Explanation : This problem seemed to stem from the f^ct that 
state evaluation guidelines were not clear on at least two points; 
(1) who at the state level was responsible for working with local 
administrators and teachers to see that recommendations- of the 
evaluation team were carried out, and (2). what mechanisms were to be 
used to let state planners and evaluators know what recommendations 

s 
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were implemented and what were the results of that implementation? 
Communicating the recommendations of the evaluation team Jzo those 
responsible for their implementation at both the state aad jlocal 
levels was also a part of this problem. J . 

i . ... 

Results : Alabama evaluation staff developed a system for 
follow-through on recommendations of the evaluation teams and 
started to test that system in school year 1978-79. Briefly this 
procedure involves a visit by an evaluation staff member and several 
subject matter specialists to each school a few months after the 
school has participated in a team review. This visit has two 
purposes:? (1) to receive a report from the local administration on 
what the school plans to do to implement the recommendations of the 
review team, and (2) to determine assistance needed from the state 
staff. Early^tests of the system showed varying levels of success* 
Some schools had given -a great amount of thought as to hbw 
improvements in vocational programs could be implemented while other 
schools had given very little consideration to the problem. 

After the year's trial period a more formalized procedure was 
developed. The major points of the newly implemented system are the 
involvement of the state director of vocational education and a 
statement of very specific prQcedures to be followed to ensure the 
effectiveness of the follow-through process. " 



Problem 3 : Communication and cooperation between the evaluation 
unit and the planning unit and between evaluation and 
the Management Information System (MIS). 

Explanation : It must be kept in mind that organizational units 
do not communicate or cooperate — people do. When project staff 
talked with individuals in the MIS and in planning , it was found * 
that they had little knowledge of what the evaluation section was ^ 
planning ory what would be required of them, This was entirely 
understandable in that the evaluation staff was^ still not sure what 
the needs would be or what evaluation could contribute to planning. 
The need to keep significant others informed was pointed out to the 
evaluation staff by the technical assistance project st$fcf in person 
and by written report. State staff agreed with the project" staff 
but questions of how this was to be accomplished came from the state 
director and those working in the units involved (planning, 
evaluation, and MIS). ^ V 

Results ; Very little progress had been made in establishing 
full camriunication ^between these three functional entities in the 
vocational department/. Project staff recommended that if the , 
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informal communication process failed to work,, a more formal 
procedure should be installed. This procedure is illustrated in 
Fig(ure 10. This figure shows how a series of requests for specific 
information with critical timelines and a method of response can 
provide essential information to all parties; the state director 
plays a critical role in monitoring the flow of requests for 
information and responses |o those requests. The newly appointed 
state director of vocational education h^s indicated that this, and 
other evaluation problems yet unsolved, will be priority areas for 
his efforts in the department. ^ 

Problem 4 ; How to assure communication on changes made and results 
observed. 

Explanation ; Th^>se directing the evaluation effort need f 
feedback on which recommendations have been implemented and the 
results of -the actions taken. This information is' needed for 
several reasons. If service to schools is to improve through the 
evaluation effort, the results of the recommendations made by its 
teams must be known. If no improvement resulted from implementation 
of recommendations, then changes must be made in teams and 
recommendations. 

The evaluation system needs proof that it is in fact bringing 
about changes which result in impro>/fed vocational training for 
students. Like all other parts of the organization, evaluation 
needs evidence that it is reaching its objectives, and feedback on 
improvements recommended and implemented is an important part of 
that evidence. 

Another reason it is important to know about results of 
recommendations and assistance is that such feedback information^ can 
be used in making state level decisions. By compiling statewide 
information, the evaluation unit is in a position to recommend where 
expenditure of support- funds could be expected to produce the 
greatest remits. Administrators face many demands for funds. 
Information which identifies the most common needs and most 
effective treatment will be of great assistance in making those 
decisions. 

0 

Results ; The procedures outlined in the paper entitled 
"Technical Assistance Responsibilities of State Staff Related to 
Vocational Program Review" were an attempt to establish a system to 
alleviate this problem. This plan clo§ely follows the 
conceptualization shown in Figure 11. Information on programs is 
fed from the local school td the evaluation section by both the 
information system and the review team. Recommendations flow back 
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to the school but also to the field representatives of the state 
staff* . Assistance is provided by the field staff, and the results 
of the changes are reported to the evaluation staff, A second 
review of local effort is seen in the annual application which the 
local school submits. In the application the school is required to 
indicate actions taken in response to the { evaluation 
recommendations. 



Problem 5 ; Improvement bf evaluation instruments. 

Explanation : , As in any new system, there is always a need for 
t revision and improvement. In fact, the purpose for testing the 
newly developed procedures and instruments is to identify changes 
that need to be made. . Several problems with the original 
instruments were identified. These problems included the following: 

1. The self-evaluation instrument was. f too long .and detailed. 

2. The supervisory instrument required a "yes 11 or a "no* answer 
to all questions when some other response might have been 
more appropriate. 

3. Some questions were multiple questions and, therefore, -very 
difficult to ^answer. ' 

# 4. Instrument^ did not provide sufficient information on 
f special services for special need^ populations . 

^ ■ > 
5. The relationship between program standards and evaluation 
instruments was not always clear. *-* . 

* / ■ 

Results : All instruments used in the evaluation have been 
B revised. Project staff, along with individuals who used the forms 
or responded to the forms, made suggestions as to how they might be 
improved . 

The following aire criteria used in revision of the instruments: 

1. Every question must gather information critical to making 

v decisions or reporting. ^ 

■ ) ^ 

2 V Any questions not getting uniform or clear response must be 
j * rewritten. . • ~ 
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3. Questions with "yes" or "no" answers must be avoided* 

4. Questions requiring judgments should have a scale to 
allow for several levels of answers. 

5. Each item must ask only a single Question. 

6. Questions on services to special populations should coyer 
, access., participation , process, and outcomes of programs. 

7.. A system of numbering should be developed to match the 
system used in the program standards^ designations. 

In the case of the instrument used by supervisors to review 
programs the n ye$ m or "no" response was changed to allow for a 
three-level choice. All questions were reduced to a single item 
response. A series of questions dealing with accessibility , 
practices , and outcomes for special needs populations Was beeii 
provided by project staff. These are being reviewed to select those 
that 'will be included. An interview guide has been developed to be 
used by team members in interviewing individuals. 

I^artijtular attention has been given to assuring that instruments 
collect information needed to evaluate programs and services for 
special populations. Questions that indicate access , participation, 
services, and outcomes of programs for all categories of special 
populations have been included in evaluation instruments. 

Problem 6; Changes needed in. the postsecondary evaluation ^^stem. 

Explanation ; The postsecondary staff had one 'major question: 
Does the accreditation done by the Southern Association of the 
Colleges and Universities meef the evaluation .requirements of the 
federal legislation and the state vpcatxonal education department? 
They were aware of the need to follow up students. This was being 
done- in the technical colleges but not in the community colleges. 
Postsecondary staffwere in the process of preparing to follow up 
all vocational studehts in postsecondary vocational education. 

* At -the suggestion of the postsecondary personnel, the National 
Center project staff prepared and presented a listing of what was 
required by the federal legislation in the form of evaluation. 
After review of these requirements , it was concluded by the 
postsecondary division that at^least minimum requirements were 
being met. - < 



68< 



c 



The project staff recommended that those in charge of ' 
postsecondary vocational education give careful consideration to the 
development of a more comprehensive evaluation program* 

. \ , . • - . . • 

* Results : The administration of the Department of Education has 
given the Vocational Research Coordinating Unit responsibility for 
the development of an evaluation system for postsecondary vocational 
education. A representative advisory committee has Been appointed,, 
a proposal to develop a program review system has been submitted to 
the Department of Education, and a tentative plan hps been 
developed. This proposal, if approved, will provide resources for 
the development, testing, and revision of a model postsecondary. , 
evaluation system to be implemented. This plan, if implemented, 
will correct the problems, identified in the postsecondary evaluation 
Ji system by the, National Center project staff. 

Problem 7 : Evaluation of services for special populations. 

Explanation : How c$h the state evaluation system provide 
information to state and federal leaders as to how effectively the 
special needs of disadvantaged,, handicapped, those with limited 
English sneaking proficiency, minorities, women, and other groups 
are being met? Additional services,, should be considered as any 
activity designed to assist special groups to succeed in a regular 
vocational program. 3 , 

, Four major elements of this problem which evaluation, with the 
assistance of other units, should, study are access, participation, 
process, and outcomes. Knowledge needs to be gained about the ^ 
extent to which policy, social, and physical 'barriers have been 
eliminated. Evaluation must be aware of the special services 
provided to assist these t individuals. How successful the special 
activities have been in reaching the outcome objectives set for all 
students -must alsd be determined. 

r 

Results ; During xhe^cbrrent year of the technical assistance 
project , efforts have concentrated on helping states to improve 
their evaluation system for special populations. Through 
observation, interview, and analysis of documents, project staff 

members have identified specific problems which need action. 

..... ^ 

Alabama has responded to the statement of these problems and 
the accompanying recommendations by a total review of those sections 
of the evaluation system relating to special populations* Interview 
outcome* guides which are used by evaluation team members now include 
a section on recruitment, entrance requirements, and removal of 
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barriers (physical, societial, attitudioal) for special populations. 
The information system is being modified to include information on 
the- instructional setting in which education is provided. Interview 
outlines have also been revised so that teams will be encouraged to 
consider the quality of instruction, guidance, counseling; 
placement/ and follow-up services for special populations. 
Attention is also given to the capacity and condition of facilities 
and equipment provided for special population individuals. The one 
weakness identified in the results section (longitudinal follow-up) 
has not, at this time, been corrected. 

■ ' * ' .... ^ - ; ' ' \ - * . 

Problem 8 : To clarify the relationship between self-evaluation,' 
supervisory visits, and team visits in the^ evaluation 
scheme. 

Explanation : These three separate activities have the potential 
for causing confusion or conflict within a program unless some 
guidelines are established. These guidelines should specify the 
role of each activity and outline arrangements for close 
coordination between and among them.' Early observation by project 
staff and .others indicated that findings during the self-evaluation, 
the supervisory visits, and the team visit could be quite different 
or even in conflict with each other. The idea of varied inout into • 
evaluation is to present different views of how programs shOulcl 
perform and how they can be improved* The advantages of these 
different views may be los^iif local educators become confused or 
perceive different evaluators as giving opposing recommendations. 

* * 

Results : It was determined by state staff in Alabama that it is 
essential to have broad involvement and input into the evaluation 
process. The danger of confusion and conflict was also recognized. 
The alternatives adopted were (1) establishing a mechanism whereby 
each succeeding*evaluation effort (selfT^ey^tration, Supervisory <^ 
review or team review) would receive tta^esults of preceding 
evaluations, (2) providing arrangements for follpw-up meetings 
between supervisors and local jteac^hers , (3) having a staff member^ 
serve on each teani, and (4) conducting orientation sessions by 
evaluation staff to clarify roles ahd expectations. 

* The 'state evaluation st^ff felt that this procedure would 
maintain the diversity of viewpoints without creating confusion* v 
Observations indicate this isN^ partial solution to the problem; 
however, further clarification of the-role and responsibility of 
each of the parties is needec^. 




Problem 9 : To provide evaluation services with 'limited resources. 

Explanation : A question uppermost in. the minds of evaluation 
staff in Alabama , and in the other participating states as well, is 
the method, to be used in providing evaluation services to the large 
schotol systems or tp the large number of programs in the states • 
Some of the large city school systems in Alabama may have as many a$ 
160 vocational teachers. This number is even greater in some other 
states. The prospect of putting together enough teams to review 160 
programs with the resources available seems impossible* The law 
specifies that all programs must be evaluated over a five-year 
period of time. * 



Results; To metet- the above problem, Alabama hag made several 
adjustments in its evaluation system. . The teaiA evaluation limited 
to traditional vocational programs. Prevocational- and career 
education programs not are included. The time, the team spends in 
the larger schools has been extended by 1/2 day. Th$ numbet; of 
former students and (employers interviewed has, been reduced, and 
where possible additional team meirfbers have been included to a 
maximum of nineteen. The System was able tb complete an evaluation 
of one vocational system in a large city last year. 

. ■ ■ * * ■ ■ 

Colorado . 

^ ~" ^ • . * 

Colorado's supervisory staff is responsible for a, portion of the 
evaluation scheme which consists of a checklist of strong and weak 
points of programs visited. \ Following the supervisory evaluation is 
a Comprehensive Program Review (CPR) . This reviev? is done by a team 
of individuals representing many segments of the* society having an 
interest in vocational education. This team has the supervisory 
findings as background information before it begins its review. 
These form a basis for recommendations on program improvement. 

Based for the most part on observation and discussions during 
several state visits, the project staJ^f felt that Colorado has many 
of the essential parts of an effective evaluation 'system. ' The 
Management Information system (MIS) can provide required information 
in whatever form desired for the evaluation effort. There* is ah 
excellent working relationship between those responsible for the 
collection, storage, and retrieval of information and those charged 
with evaluation. The range, quality, and availability' of data and 
the cooperation of the staff are strong contributors to an effective 
and efficient evaluation system. 



Problems 



' The technical assistance team in* conjunction with the state 
staff identified a number°of problems in the Colorado evaluation 
system. These problems were presented to the state in a formal 
written report, in a meeting with evaluators and administrators, and 
in£pEmally , to individual members of the state staff. The problems 
presented for consideration and action by the vocational department 
were the following: 

1. Measuring student achievement 

2. Increasing LEA self-evaluation • 

3. Securing employer follow-up 

3A, Assuring confidentiality of employer follow-up 
3B. Securing continuing- cooperation of employers 
4V "Using evaluation information and data in planning ■ 

5. Identifying schools/programs to evaluate . k • 

6. Systematizing all evaluation effortst 

7. '^^Determining effectiveness for speci^Q needs. groups 

8 4 . Developing standards for program evaluation ' , 0 
9. Meeting requirements of mandated postsecondary evaluation 
10. Increasing effectiveness of the supervisory review 



Whiie several of these problems, were acted upon in/a positive 
w^y, the state, in compliance with a legislative mandate, made 
extensive revisions in its tot^l approach to evaluation., This 
restructuring Of, the^ evaluation system' occurred simultaneously with 
(and as part of) a complete Reorganization of the vocational 
department. 



:omplete & 
specified 



The legislature specified that the state vocacibnal agency. 
/ should first identify the least efficient and. effective 20 percent 
of its programs, and secondly, concentrate its progf am improvement 
efforts on these programs which^had the lowest performance score'. 
kit may be added parenthetically that tBne -Natiohal Center project 



team had for some time been urging that one or more states test this 
approach. Fortunately , Colorado had a^data base which made it 
possible to compare programs on a number cost and outcome factors 
and to arrive at a logical list of those programs most In need of 
assistance* • / 

In the Colorado model , efficiency is define.d as local cost per 
full time equivalency student (FTE) as a percent of'.srtate cost per 
FTE for a particular program* Ef fectiven*ss is defined as the 
ex tent, to Which state goals are met by the local program. These 
goals expressed in terms, of criteria are completion rate of 
enrollees, placement rate of completers , enrollment of minorities , 
enrollments of handicapped , enrollments of disadvantaged 9 and sex 
balance. Performance in the Colorado procedure ifieans effectiveness 
in meeting goals relative to the resources expended oh a comparative 
basis. ' ■ ■ .' ' * 

The first ranking of programs has been generated and distributed 
to schools. The resgbnse by schools to v this procedure was reported 
to be very positive. * The next step in the process will be to give 
the subject matter specialists (supervisors) a summary report on 
each of the low-performance programs. Supervisors consulting with 
the local program and school will report back to the NState on the 
reasons the program ranked low and what the school proposes to do to 
improve the situation. Programs found to be in the bottom 20, 
percent a second year will have a state review to determine whether 
the program should be terminated or receive further assistance. 
This experiment should be watched closely and its progress reported 
tp the evaluation community periodically. 

\ . 



Maine 

The Maine vocational delivery syst^fl consists of 
twenty-five school systems providing vocational education.. 
Each of fourteen <>f these are administered by a single high 
school with responsibility for serving aJU students in the 
district. The eleven additional schools are administered * 
jointly by a number of schools in various regions. These 
centers, or regional vocational schools, are evaluated on a 
five-year dycle which includes an extensive self-evaluation and 
team review. - 

The project team visited the state several times to become 
acquainted with the evaluation system, to identify and discuss 
problems, and to review the present data system. -The teiam 
observed a team review of one of the regional vocational 
schools. J 

The state choose not to continue with the technical 
assistance project in Year II. Since an alternative state was 
chosen, contact and follbw-up on the solution of problems in 
Maine has been very limited. Following are the major problems 
identified and the efforts made to correct these problems 
during the first year of the project. 



Problem*,!: Altern#tive to Self-Evaluation-Teart Accreditation. 
* 



Explanation: State staff and. local educational personnel 
have expressed concern over the amount of time required for the 
self-study which precedes the team review. It is estimated 
that a school may spend as much as seven months in preparing 
the self-study report. While this was reported to. be useful to 
the school, it was thought that a five-year cycle was too often 
to do t^is* in-depth analysis. Having heard this comment from a 
number of schools, the state staff accepted ifhis opinion. 

In attempting to make the total evaluation effort ^ts 
meaningful and productive as possible, the Vocational ) 
department staff as. well "as the state advisory council staff 
have expressed an interest in. exploring options to repeating 
the self evaluation-team cycle every five year^. 

- ■ * 

Results : 'State staff are now considering using in-depth, * 
teacher personel interview follow-up of former students and 
employers as the alternative system.. This would gather 
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information in much greater depth than could be collected in an 
annual student follow-up. 



m Project staff were, requested to propose procedures and 
sample quest ins to be used by teachers to interview former 
vocational students and employers. The system proposed by the 
project staff would train teachers in interview techniques, 
provide an Interview guide for collecting information f rim 
former vocational students and employers, and suggest ways of 
reporting and using the information. The project staff 
proposal was reviewed by state staff and its adoption was 
recommended to the state board. This was the situation at the 
time of termination of the project. 



Problem 2 : Student Follow-up. 

Explanation t The state conducted a test follow-up in three 
schools during the 1977-78 school year. This follow-up was 
conducted by lending follow-up forms to students from the state 
through the local schools. Forms tfere 'returned ta^the^ l&cal 
school and then transported^ to the state office for computer 
analysis. Follow-up letters and phone calls were usecfr to 
increase the response rate. Results of the analysis and the 
origitial forms were sent back to the local school. This 
process .resulted in a 55 percen.t return rate f rom jprogram 
completers. Former students were asked to give approval for 
the staff to contact employers for additional information. 
Eighty percent of the responding students answered positively 
to this request. Employers were th#i sent a questionnaire to - 
asqertain their impressions of the level of training of the 
employees who had participated in vocational training. A 74 
percent response rate was realized through the original 
^mailing, a reminder letter, 'and a phdne call. 

* * ■ 

Results: At this point, project staff reviewed results and 
procedures and made comments and recommendations. In the 
review of the follow-up instruments, several* suggestions were 
made which were aimed at assuring more accurate analysis of 
responses, ^bme changes in procedures, as outlined below, were 
also suggested. As a general guide to revision and improvement 
of the evaluation system, the "Handbook of Followrup of 
Vocational Students,", developed by another project in the 
National Center's Evaluation Division, was used. Specific 
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suggestions were (1) reduce the responses allowed on questions 
where multiple responses might be confusing to the* analysis, 
(2) change wording of some possible responses to make thein 
correspond more closely to the questions, (3) divide some -items 
where more than one question was asked, and (4) change some, 
procedures to require less, instrument handling. It was 
stressed 'by project -staff that a special effort be made to 
check a sample of the non-respondents to see if they appear to 
be markedly different from those responding . 

NewlyVadppted procedures include the mailing of 
questionnaires from a* central location and the return of the 
completed* form to a central location. Cbmputer analysis will 
be completed, and the results and original instruments will be 
returned to the students 1 school. The letter accompanying the 
instruments will have the name of a local official of the 
school from, which the student graduated. 



Problem 3 : More Effective Use of Evaluation Information 

Explanation: The use mad^/of the information generated is 
the real determinant of the benef ifcjs of an evaluation system. 
Evaluation 'data and information can make a contribution to the 
decisions made at both the state and local level. 

, : This problem encompasses both the use to be made of the 
information afttqi> r the form and, procedures in which the evaluation 
findings may best be jpresentepl. Three questions relating to 
this overall problem were considered. First, how should fche 
^information be analyzed and packaged? Second, how should the 
information should be presented to schools? Third, how can 
this information be u?ed as a basis for decisions? 

Results;, Question 1 - How should the information data be 
analyzed and packaged? • 

* 

Presently follow-up dat$ showing number and percent of 
students who answered in the several categories are provided to* 
schools with no comparisons. The* project 'staff recommended that 
in the coming year, the state consider the calculation of an 
outcome index. This involves reaching ^a consensus on desirable 
outcomes, assigning a weight to each outcome, and developing a 
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formula and computer program to do the calculation. This 
reduces all of the expected important outcomes to one figure 
and is a method o£ indicating the relative effectiveness of 
programs in achieving those outcomes considered to be 
important. ' 

Results; Question 2 - How should the information be 
presented to Schools? * » -» ^ 

The state will compile follow-up data from students and 
employers and provide this information to the schools and the 
visiting team. Teams will present major recommendations to the 
school at an exit interview and will provide a report to the ^ 
school at a later date. . \* 

Results; Question 3 - How can the information be used as a 
basis for decisions? 

Possible uses' at the state include the fpllowing; 

* ^ 
\. Basis fdV decisions relating to expansion, 

continuation, or termination of programs 
2. Guide for efforts subject matter specialist 
(supervisors) in assisting schools/programs to 
improve. ' , \ / 

3# Guide to ^f und allocation for supplemental services. / 

4. Indicators of inservice and preservice education 
needs. ^ 

5.. Indicators of impact x&n vocational education. 
•6. Indicators of effectiveness of services for special 
needs groups, |M / 

Possi$L# uses at the local level include the following; 

1. Indicator of changes needed in school a^d program. 

2. Indicator of impact of .vocational education.* 

3. Guide to fund allocation. 

4., Indicator of pro^r&ms needing special assistance. 

5. Indicator of effectiveness in meeting needs of special 
needs and individuals. - / . 

Obviously, bo^:h local and state decision makers should be 
ti^in$ the results of evaluations as an input injto managing the 
vocational education program. Too often these results are not 
used pattially because, the manager is not aware of the 
potential benefits of this procedure and partially because 



evaluation findings are not packaged in the most usable form. ' 
Project staff have not been able to determine if there has been 
any increase in the use of evaluation information. 



Problem 4: Evaluation of State Vocational Delivery System. 

Explanation: All states are engaged ijn some type of 
evaluation of local programs. This consists mainly of 
answering the question "Are we doing things ri%ht? w Few states 
are concentrating on evaluation of the state program of 
vocational education and attempting to determine " Are we doing, 
the right things?" The accountability report specified by the 
Congress at least to some extent^asks this same question of the 
states. ■ 

To evaluate effectively a state must ask itself the hard 
questions: * 

o To what extent are we meeting the manpower needs of the 
state? ~ 

o Are programs accessible and open to all who could profit 
from training? 

o Are we providing training for the most critically needed > 
occupations? 

o Are we providing training in which there is adequate" 
w for training and expectations of advancement? 

o Are there opportunities for retraining of employment ^f or 
every student, enrolled in vocational training? 

These 'ancjl many other questions could make up, the 'score 
sheet against which the state program fcoultl be judged. 

Results: At present, the accountability report will be 
used to evaluate the Estate's program. It was recommended by 
the project staff ^fc^t Maine oonsider th* formation of a 
consortium of statqBincluding vocational' staff and SACVE 
members to perform tj|i evaluation of the state vocational 
system. ^ * * ' ■ ;° • ' 
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Problem 5 : Follow-through on recommendations of ste^f-study 
and evaluation team. 

Explanation: Probably the most important function of an 
effective evaluation team is not to evaluate but to' make 
recommendations f6r improvement. Schools need- assistance not 
only in . identifyibg problems, but in solving .them. A 
follow-through system on evaulation recommendations. is 
essential if maximum benefit is to be realized frpm evaluation. 

This follow-through consists, of several important phases 1 
First, the state should be in apposition to provid^ assistance 
in implementing recommendations. Second, the state should 
require that critical program impjjgvements be made if programs ' 
are to continue. Third*, there should \& feedback to the 
evaluation unit on how recommendations were dealt with, it is 
important that local arid state staff clearly understand who has 
what responsibility in this e'ffort to help schools arid programs 
provide better vocatfona} education.' ". ^ 




Results: Maine is placing major responsibility fpr 
assistance with^the sj$pervisory staff and including reports on 
improvements in the State plan. ' * I 

Two additional actions were recommended to support pnograip 
development. First, as state summaries of evaluation are made, 
a compilation of recommendations will show the improvements 
that are most Commonly heeded . The state could theri allocate 
resources (dollars> staff., etc.) to the- solutiohs of theslp most 
common problems. Second, the vocational teacher education 
staff should be made aware of deficiencies identified in 
schools sp that pr^servi^e arid inservice programs could stress 
these points. 

Problem 6: - Identification of outcomes (other than placement) 
for which vocational education should accept 
V r responsibility cfnd take credit. 

.. E-xfoanation: Occupational' 'placement has always been an 
important objective of v.Qcatibnal education. This will 
continue to be important both as a measure of effectiveness an 
as a guide for updating and improving o prj8grams. Thera are, 
however, other outcomes of vocational ^education. ^Programs and 
schools should be aware of these other results aiS|^Ctempt to 
increase the benefits that students 5 receiye. Vocational 
educators should also strive to identify ways of measuring 



other outcomes as an aid in evaluation and as a credit to 
programs. * 

This problem was especially important to the executive 
secretary of the State Advisory Council for Vocational 
Education. The position was that vocational education, 
especially at the secondary level, should not get trapped in 
considering the only worthwhile outcome to be placement on a 
job. v 

Results: < No action has been tadcen at this time, but 
recommendations are under consideration. The project staff 
have recommended that input from the National Center project 
"Examining Vocational Education Outcomes" *be bbtaiped before 

ther action is taken. Interest has been expressed _i/h the 
ssibility of calculating a product index for each program, 
is product index would give weight to other outcomes in 
dition to placement. * 

j 

Problem T; Determining -effectiveness of practices and programs 
for special subpopulations (disadvantaged, 
, handicapped, minorities, limited English 

proficient, and women). * 

Explanation: There are some aspects of the problem in 
Maine which may influence the choice of solutions. 4 

J. Many of the evaluation team visits wilL be done under 

the auspicies of the New England Association of 

Schools and Colleges. The interjection of 

appropriate qustions into the review guides may pose 

more q£ a problem than a state conducted evaluation. 

♦ 

•r " p 
2. The types of special needs, the special services 
rendered, and the attitudes of the school and 
community may be quite different in different 
sections* of the state. s 



Results ; Efforts have been made to insure the state's 
ability to identify individuals falling within these "special 
categories iri^enrollment and fallow-rup procedures. 

✓ . . . * • 
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Problem 8: 



Coordination of evaluation with accreditation by 
the New England Association of Schools, and 
Colleges, 




Explanation: ^ K cooperative agreement has been worked, out 
whereby the local school will use^ the program review ("P M form) 
.developed by the state for self-studyl prior to its review by a 
\eam selected by and representing the New England Association. * 

Project staff had ai^ opportunity to observe one! of these 
visiting teams reviewing an area vocational school. Two 
problems were identified with this arrangement. First, team 
members and the Association seemed very reluctant to have othet 
individuals observe or participate in the process. These 
•"other'" individuals were representatives of the State < 
Vocational Departirtent, members of the State Board for * 
Vocational Education, and State Advisory Council personnel. 
Second, team members seemed unable to recognize as acceptable 
any arrangements for service to students with vjhich they were 
familiar. Since this was the first of several" planned team 
visits to several schools, it is hoped that the Association can 
^encourage- morel flexibility in its team member s*. 

The state is continuing to negotiate with representatives 
ot the New England Association on modifications they feel 
should be made. Progress is reported in defining the role and 
responsibility of the ex officio members the^ state feels should 
bef involved in "the review teains. \ v ■ * 

Maine is a relatively small state with limited staffr to 
perform functions of the vocational department including 
evaluation. Problems for which immediate sqlutions are being 
wo r Wed bOtare the implementation of a follow-up of students 
and employers and coordination of evaluation with the 'New 
EnglaQd^ Association of Schools and Colleges, The installation 
of the^ai^er native five-year evaluation system and solution of,, 
other problems identified in the report are long range 
objectives of the department. 
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California 

- : "( 

The secondary evaluation system in California is » 
responsible for (1) describing the statusf of the program 
delivery system and (2), providing information on sources Of, 
support or information on problems. Thfs description of status 
could iead to the setting of some state priorities for 
assistance to programs. .The focus of evaluation in the 
California Community. College system is on effectiveness of the 
local commdnity college delivery systenu ,Ii> the past the state 
has provided incentive projects for addressing major needs 
identified by the evaluation syfetem. 

Problem 1; Collection of data in th£ most efficient manner. 

Explanation ; Both the secondary and postsecondary 
evaluation systems have responsibility for collecting and 
analyzing follow-up data and information on vocational 
completers and leavers and employer follow-up. The 'amount of 
#ata involved in this process in California makes it imperative 
that the most efficient methods be used to collect and analyze 

information. . 

/ ■ . • * - 

Central to efficiency in data collection is a clear 
specification of what data are essential to effective 
evaluation. At this point in time the Vocational Education 
Data Systems (VEDS) is attempting to identify critical data 
elements; however , there* still appear to be some problems Vith 
VEDS, particularly at the postsecondary level. The Office of 
Civil Rights (OCR) guidelines are also requiring data and 
'information from the states. 

If at some time ^Ln the future the state could identify the 
outcomes expected of vocational education and initiate a system 
of measuring outcomfe objectives , a guide to data needs would be 
provided. Individuals representative of all affected 
institutions should be involved in specifying the outcomes to 
be required and the data necessary tb determine the extent of 0 
achievement of each objective. 

Results: Secondary vocational education has developed a 
procedure for Follow-up of Students and Employers (FUSE) . This 
system, as originally proposed, would provide guidelines to tlte 
local' institutions on how to collect and report the necessary 



information. This plan is now being revised to minimize the 
data burden on the local school. Study is* continuing on how 
the necessary data may be obtained with a minimum input of 
resources* 

. \ 

Postsecondary vocational education in California developed 
the Student Accountability Model (SAM) . This cooperative $ 
effort among community colleges , in conjunction with the 
Community College Occuational Programs Evaluation System 
(COPES) , should provide the data and information necessary for 
evaluation in as economical a manner as possible. 

*« 

Problem 2; Targeting the evaluation system, ? 

Explanation; A question asked in connection with ^ 
increasing the efficiency of the evaluation is "Are we 
targeting on the right audience? Are we /Asking the right 
people the Aght questions?" Evaluation information must be 
able to describe the services and determine program 
effectiveness for a specific set of individuals and identify 
ways programs need to change in order to increase 
effectiveness. This set of individuals may be all students 
participating in a program or some special group. Pi^pgram 
description and effectiveness for regular students may carry 
with it a different set of questions Tthan those .concerns about 
program description and quality for speqial needs populations. 
It is incumbent upon evaluators to be able to specify the j 
target client and the means of determining services and * 
effectiveness for the various groups vocational education is 
responsible for serving. States face the choice of either one 
evaluation system for special as well as regular* students or 
separate systems. 

Results: The decision has been made to target one 
evaluation effort at all groups to be served by pro^a^ms. Thus, 
one evaluation wij.1 describe both regular students and special 
needs students being served and services provided. 

The National Center project staff provide an analysis of 
the ability of both the secondary and postsecondary systems to 
provide thfe data and information fteeded tp evaluate for special 
populations. This analysis identified ways in which the 
existing system should be modified to) ensure an evaluation 
which was truly reflective of program' effectiveness for all 
students. 5 , 

, •• ; . ' - ' 
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The secondary evaluation coordinator submitted^ 
vocational education executive staff issue -'memento the* 
administrative staff of the vocational education department. 
This memo discussed each of the problems identified by the 
project staff, suggested action alternatives, and recommended a 
solution. The executive staff, in turn, accepted the report 
and approved of the recommended actions. This action * 
demonstrates the department's support and 9 determination to 
improve programs and services for special populations. 

The postsecondary evaluation system in the form of COPES 
and SAM was reviewed by the ptoject staff and needed changes 
were pointed out in a written report and in person with the , 
evaluation coordinator. This* information was considered in the 
revision of the procedures and instruments to ensure tHat the 
evaluation system is targeted to all of the clients which the 
occupational community college system is Serving. A 
longitudinal follow-up of community college students developed 
and presently being tested in the state shows -great potential 
for expanding the data Available for evaluation and can provide 
valuable guidelines to other states considering follpwing 
program completers for an extended period of time. .■ * \ 

. . \ . 

4 

% 

Problem 3; Determining the effectiveness of the evaluation 
system. 

Explanation; Administrators and evaluators at- bo.th^the 
secondary and postsecondary levels have 'requested an assessment 
of s their evaluation system. Those persons on state staff have 
expressed a desire to have the National Center project staff do 
a criti'que and evaluation of their system. Information has 
Jaeen requested on/(l) the extent to which the present system 
meets *the feder^J: requirements, (2) the extent: to which the 
present system is meeting state and local needs for program 
improvement, ^hd (3) recommendations for improvement. 

Those in administration and those in evaluation seem 
genuinely interested in improving the evaluation system, and 
the request for, an a&sessment appears to stem from that 
interest* in improvement. Administrators are understandably 
anxious about compliance while evaluators naturally are / 
concerned with the complexities of making their plans 
operational. _ 

o , . 

i/. . % • ■ ■ 

Iftesults : No decision has been made on a method for 
evaluating -the evaluation system. National Center project 



; 4 . ■ 

\ 93 



o 

ERIC 



85 



94 " 



7 



\staf f recommend that evaluation by users be considered- A 
survey of these user groups could provide a checklist of 
exp^tations against which to judge evaluation efforts. \ - > ^ 
Project^xaff in conference with administrators of stats * C 

education agencies identified the following items as being 
important in evaluation: " 

1. Evaluation shbuld be based on other benefits of 
vocational education in addition to. placement. 

2. The evaluation should go beyond compliance and look at 
state needs. ' > 

A. Evaluation should identify those programs that need ■ • r 

help and suggest ways they can improve* , - 

4. The state should develop a solid data system. ■ x 

5. The data along with evaluation should be used as a 

basis" tot allocation of funds. T 

6.. Evaluation should provide evidence that dollars spent ' 
in vocational education produce^ more payoff thanx 

* dollars spent elsewhere. Payoff in terms £f training 
for jobs and making people employable should be 
reported. * 

7. Evaluation should provide solid Evidence that 

vocational education makes a dif reren'fce in jobs, pay^ 
and upward mobility. 

6. Evaluation should provide evidence to* us^ih 
eliminating ojr redirecting programs* 

9. Evalua^M> n should describe program results in terms of ' .* 
placement* ~ ** *^ 

10. Evaluation should detetmine competencies achieved by 

• students. • 

11. Evaluation should determine how well teachers are doing . 
in terms of student reaction , . updating of course 
materials , contact with industry and what students do 
with training. 

While thesg expectations are certainly challenging, and* 
perhaps in some instances uprealistic, with the requirements of 



* 

the other users added, this could provide the idealistic 
yardstick . against whiph to measure the evaluation systejn. 

r During the second year of the technical assistance project 
an assessment of the state's abiiity 'to evaluate its services 
and programs' for special populations has been done* This could 
be viewed as an evaluation of the evaluation system as it 
relates to that segment of the programs of fered. + 

f 

Problem, 4 : Coordination between -secondary and postsecondary 
evaluation system. , 0 

Explanation; In assuring efficiency and effectiveness of" 
the evaluation system, it ,is essential that there is close 
coordination between the secondary Jand postsecondary work. 
Many will agree yrith this assertion; however, the difficulty 
comes when practical implementation of this concept is 
attempted. An important point to keep in mind is that systems 
and organizational units do not ^coordinate and cooperate— * 
people dp; If good .working relationships are established, it 
is because people have made the effort and adjustment's/ 
necessary in any such situation. 'As a minimum requirement, 
the data elements and definitions should kje comparable, and 
effectiveness of progratos should be mutually defined. 

Results; California is an excellent example of* a highly 
effective cooperative effort between the evaluation systems 
serving these two levels of vocational education. Given that 
fact that each level has its own unique needs 'and expectations 
the two systems are compatable and comparable in all po&sible 
aspects. Based on observation, tlie project team cbns-iders this 
an Outstanding example Of coordination and cooperation and 
exchange of information and ideas between individuals* 
responsible for evaluation at the secondary and postsecondary 
levels in California. , 

Problem 5; How to do employer follow-up. 

Explanation; It is important for teachers and decision 
makers at state and local levels to know how employers viewthe 
adequacy of training of recent vocational program completers. 
This, along with a number of inputs, prtovides varuable 
information on how well programs are reaching ^student outcome 
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objectives and on, changes which need to^be made in programs. 

' • s * • .47 

This information may be difficult to get. Employers may 
feel that, this is an infringemerit on their time and refuse to 
respond. There may be concerns about the confidentiality of , > 
this information in the minds of employers, in larger 
businesses, the inquiry may go to a person who has no knowledge 
of this employee. The collection of adequate, reliable data 
could be very expensive. < v \ 

In spite of these difficulties, it is important thht Kays 
to gather this information be explored. First, the Education 
Amendments of 1976 mandate that employer reaction, to training* 
be used in evaluating programs. Second, this information 'is 
valuable in bringing ab^tft program improvement. Third, this 
information can be used to convince others of the value of 
vocational education. Fourth, this survey can be a factor in ' 
strengthening relationships between vocational education and 
the business community. / 

v •. • ' . 

Results: The FUSE system designated tb provide follow-up 
data from secondary students and employers was discussecl 
earlier in this chapter. The major discussion point seems to 
be whether the dtate <or the local .district will assume major 
responsibility in actually collecting the data. Once this 
question is resolved, the planned procedures should be adequate 

to obtain the needed information. 

v * 

The community college vocational evaluation staff have 
expressed some serious questions sabdut the validity and 
usefulness of employer follow-up information collected in the 
way VEDS proposes. Before initiating^ an employer follow-up, the 
, system is waiting until more definitive information is v provided 
by VEDS . 

Problem 6; Securing commitment of administration, field staff, 
and local schools to evaluation. 

Explanation: Administration and the field staff .appear. to t 
be committed to evaluation- They S"till have some questions as 
to whether the present system is the one which is best for the 
state. Local schools will have reservations about any activity 
which requires their time until they- are convinced of the , 
benefit to their own school or program. The problem then seems 
. to be one of demonstrating to several interested parties that 
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this evaluation system will meet their expectations. 



Results; California Efrovides an ^x^mple of the fact that 
priorities must be set when therp i& art abrupt decrease in 
resources • Evaluation planning at both the secondary and 
postsecondar^ levels is attempting to reduce its data 
requirements to what is absolutely essential to jneet statte 
needs and legislative mandates. Advisory committees are used 
extensively to.reylew plans and to, gain support. National 
Centex? project staff have attempted to recommend only what is 
critical to having an effective evaluation. These 
recommendations translated by state evaftiators, have resulted 
in- a positive response from those responsible for 9 * 
administration of vocational aducation. 
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CHAPTER VI 

• H .'-<•..' 

STATES' ABILITY TO EVALUATE PROGRAMS AND SERVICES 
.FOR' SPECIAL POPULATIONS: STRENGTHS AND WEAKNESSES 



-' , / ■ ' ... 

% This chapter presents -the states' 1 strengths and weaknesses with 
regard to their ability^ to evaluate programs and services for^ 
special populations. Ability "to evaluate was examined under four 
areas: (1) . determining program accessibility , (2) determining 
participation/ (3) identifying services, and (4) measuring progtam 
outcomes • ^ 



Alabama 



Determining Program Accessibility > 

% *■ > . ** 

At the secondary level, information* on vocational progrart 
accessibility for special populations was gathered by the < 
Instructional Program Review instrument *{ I PRI) , F.orm M 2 (Monitoring 
Checklist), and Interview Outlines. The relevant data that were 
addressed by the different instruments were as follows: 

a. The. IPRl dealt mainly ; with program accessibility in 
relation to gender, i.e., eqtial access for botji * 
male and female students and architectural barriers 
« for the handicapped. ^ 

b*. Form M2 — a monitoring checklist used by -the Division v of 
^ Instruction, Prograijj for Exceptional ^Children and * 

Youth-^contained some items On program accessibility for 
exceptional children as \t related to facility. * 

c. The Interview Outlines that were being used~ by the Team 
Review contained very general* questions, regarding the 
determination of program accessibility. 

The discussion above shows that the determination of program 
accessibility for secondary vocational programs was generally 
limited — access in terms offender and physical barriers forythe 
handicapped. The secondary state vocational evaluation system wa^ 
not gathering data and information with regard to the "following: 
recruitment procedures employed at the school level, local entrance 



requirements and local efforts to remove attitudinal aneF' societal 
barriers • , * . , 

,In addition. to the foregoing limitations, there were questions 
relative to the validity an^^eliability of the evaluation 
instruments being used, particularly the' interview outlines • 
First, criteria for rating programs were nonexistent* Second, key 
wotds and phrases which can have to multiple interpretations were 
not operationally defined • » 



Determining Participation 



> 



. r Participation of special populations was determined through 
data collected by the. MIS. All vocational schools were required to 
furnish the following enrollment data; ' 



a. Student's name 

b. USOE code 

c. *Grade level 
a/ Sexy 

-.1 



e. Ethnic origin 

f . Disadvantaged 
g^ Handicapped 



Alabama lacked required VEDS $3ata on students with limited 
English proficiency (LEP)' and enrollment by instrtictipnal settings. 



Identifying ^Services for Special Populations ( * 

information on the. services for special populations was gathered 
through the IKRI and interview outlines • The data collected 
included the following: *~ ^ 

a. In the IPRI, the data pertained to identification of - 
services for the handicapped and sex fairness/stereotyping. 

b. In the interview outlines, the data focused on the 
handicapped, disadvantaged, and women. ' 

The secondary evaluation system did not investigate the 
following elements in the process of identifying services for - 
special populations: quality of instructional offerings, guidance 
and cpunseling, placement and follow-up services, capacity and 
condition of facilities., and for special populations* 
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Measuring Program Outcomes ^ ' ' » 

The MIS collected information on completers by asking every « 
institution to complete DHEW OE Form 346-4 (Placement of Program 
Completers in VocartiortXl Education Programs} . The form did not 
contain data K and information on stud.ent outcomes as required by 
VEDS. However , it was possible to Cross-tabulate 'the information on 
completions with £he enrollment data. Alabama, therefore, could 
comply With the VEDS requirements if proper computer programming was 
used but there was no, evidence that this type of capability existed 
at ^the time of this study£ * / * 

Alabama did not have the ability .to meet other requirements of 
the Education Amendments of 1976, i.e.,. to determine employer 
satisfaction and to measure student achievement with standard 
occupational proficiency measures, criterion referenced tests, and 
other' examinations. 



California 

Seqondary Vocational Education Evaluation 

v Determining program accessibility . Information on program 
accessibility for special populations was gathered throuqh the 
Program Administrative Review (PAR) and the Program ' \ 
Assessment-Vocational Education (PAVE). The specific information 
^collected by California was, as follows: - „ , y * 

a. In the PAR instrument, the data focused on, documentation of 
* program barriers for t^he handicapped and disadvantaged 

groups. : • m ■ . * 

b 7 The teacher interview scheduled in PAVE 'examined mainly ; 
"special efforts to^ricou,rage special populations to 
enroll". and prerequisites for enrollment. The same was ^ 
true of the .student, questionnaire. 



c. The PAVE Program Self-As<§£ssment Questionnaire limited 
- was concerned with ^eqi^al access as it relates to gender . 

. j * * r l . 

The ability of the* secondary vocational evaluation system to 
determine program accessibility was, limited to fehe issues cited 
above and certain special^population subgroups./ It failed to deal 
with all the special population subgroups, particularly the 
minorities and limited English proficiency (LEJP) . further, it did 
not collect data or information on local school efforts 'to remove 
attitudinal and societal barriers # ^ . 



Determihigg participation . , In* the* |ecpnda1:y vocational ■ • 

j education t system # determining participation of special populations 

was made, through the Management Information System (Forms VEA-:48 
Sections I and II), PAR , % and <PAVE/ The MIS 1 gathered the following 
<3ata on students: , fc , ; • 

a. Ethnde classification mm e. Disadvantaged. * 

b. Grade level ^f. Handicapped ^ ' 

c. Sex - * ' ,9 s USOE program code 

d. Limited English speaking i , * : / . 

The MIS forms enable^ California to determine participation 
according to sex designation and ethnic classification in the USOE 
- six-digi^t and tw(?-digit codes. The MIS fo^tas did not allow 6he 
V state to determine enrollmentfon types- of fcandicapped, 

V7 « disadvantaged, and instructional settings as required by VEDS 

starting, with 1979-80. Further, the MIS did hot allow the state to 
determine participation of limited English proficiency students in 
the USOE six-di^it and two-digit codes. 

' r * t • ; ■ • ' ..." v _ 

Identifying services for special populations . California 
\ collected limited information regarding the identification of 

services for special population^, through PAVE, and PAR. 1 in PAR, • 
questions on process elements were mainly "for the disadvantaged and 
n handicapped. In PAVE, questions on process evaluation were general 

, 1 in nature. On the other hand/ the questions on appropriateness of 
v * instructional materials were limited to the handicapped and , - .• ^ 
\ disadvantaged. / s, 

^ The foregoing discussion shoiys that the California secondary ' 

evaluatiQn system failed to treat the identification o£*s€&vices 
across all special population subgroups , especially minorities and 
/ ' limited (English proficiency students. v \^ . . 

I - * . ■ . . 

, 1 j- Measuring program outcomes . At the time. t>f this study, 

California w;as; collecting data on program completers knd leavers 
which satisfied the requirements' of DHEW OE Form 346-4 but not VEDS 
.- w * requirements. The Follow-up qf Students and Employers (FUSE) which 
sought to fulfill the VEDS requirements (among other things,), was . 
stillvin the planning. Measuring student achievement also was still 
in the planning stage. 

- - • ■ •'■ ' . .. , ? 

, Postsecon4airy Vocational Education Evaluation 

Determining program accessibility . The California* Postse.condary 
•Evaluation ^System determined program- accessibility through its 
Community College Occupational Programs' Evaluation System (COPES)..-. 
In COPES, the data gathered were limited to: * 

' ' ' " * 
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a. Efforts to achieve sex equity in program enrollment, 
recruitment , and admission policies. ^ , 1 

b. Architectural barriers- for the haiidicapped . 

•'\'-. * * : 

The activities of item (a) should hav^f been addressed to all 
special population subgroups, i .e. , women, handicapped, 
disadvantaged, . limited English .prof icienoy, and minorities. 
Further,.. COPES failed to evaluate local school ^efforts to remove 
attitudinal. and societal barriers. 

Determining participation . The determination of participation 
of special populations has been made through the Student ' 
Accoantablity Model (SAM). The data gathered from the forms — 
especially the Student/Course Inventory form 1 — indicate that the r 
California postsecoridary^ vocational 4 evaluation systeim can gather 
important information not ^only fo.r compliance purposes but also for*^ 
planning and program improvement. At the> time of t£e .study , 
however , enrollment of the various categories of special populations 
was determined *by the USOE two-digit codes in view of the unresolved 
technical difficulty associated with the USOE six-jligit code £t the 
postseconc3ary level--a problem f recognized by VEDS* Further, no dat§ 
on \\ and i capped enrollments by instructional settings were^oj^lecT^ed . 

% Identifying services ^ COPES can gather adequate i^fornflftion on 
the identification of additional services for x special populations. 
However, in the student questionnaire, identification of^jrespondents 
was limited to college name and course title. A definite sampling 
plan was not adopted to determine the special population 
respondents. Additionally, it was difficult to isolate the responses 
of the special populations from the responses of all other student 
respondents. The sampling procedure recommended b^ COPES used V > 
course classification. . • 1 

Measuring program outcomes . SAM offered a system or examining 
student outcomes for both regular and special students not only for , 
compliance purposes for also for planning. It made it possible to 
compare and contrast regular , and special students in terms of 
program. completions, successful placement, and continuing education 
in different program areas (although there was no evidence that data 
were being collected in this way) . At the ✓time of the study, SA£1 
did not 1 include data on continuing education over time. Thd 
postsecondary vocational 'system, like the secondary, also did not 
Jhave the ability- to measure student achievement as mandated by J:he 
Education Amendments of 1976. # v 



* ' Colorado ' > . 

Determining Program Accessibility f . 

Through the Comprehensive Program Review (CPR), Supervisory 
Visitation, and ^Institutional Application Plan (VE-120 ), Colorado 
was able to collect some information on program accessibility. .The 
specific information was as follows: - 

• . &\ * ■* • 

a. \ % In the CPR, and , Self Review instruments, qjues^ions on ' x 

accessibility w^re' limited to the determination of , 
physical barriers for the handicapped. The same was 
true 7 for the Supervisory Checklists. HoweyerV not 
all\ checklists contained questions on physical 
accessibility. r 

* . . . '-.«•■ 

b. VE-120 examined recruitment and selection of students 

in t^rms of sex equity. \ .*•>'• 

Determining Participation s~ . \ • 

^he Colorado Management information System (MIS) gathered data 
on Student enrollment using form VE-135 (Student Accountability Data 
Tool). Effective* last school year (197§-79), MIS w,as collecting, * 
data on special populations according to VEDS requirements except 
data enrollment by instructional settings.^ * , - 

■ ' ■ ' x ■ " ■ ■ - • . V ' 

Idettttifying Services \ 

^ Through the CPR and supervisory visitation, the i/iformation on 



the 0 quality of instructional 
and equipment used was limite 
was confined to the following: 



offerings and condition of 'facilities 
3 to regular students o'nly . information 



a.- In. the self-reviefo instruments, focus was on the 
* disadvantaged and handicapped. The quality 

indicators were concerned with the mechanics of 
compliance rather- than the assessment of quality 
. of additional services as they affect quality .of 
products. ° ■• 

b* Some.^upervisory checklists contained questions on. spet:iaj. 
services. Questions were mainly ^concerned with the" 
assessment of tutorial services - and the use of 
instructional (bias-free) , materials . 
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The "Guidelines for th6rUse of Disadvantages/Handicapped 
Services 11 (VE-116) contained a program review checklist for 
supplemental services^ However, there were questions* on the 
validity and reliability "of the instrument, " The checklist lacked 
criteria for rating services. Furthermore, its rating stal^was not 
operationally .defined. • \ # # >' 

There *are, therefore, serious deficiencies in the identification 
o'f special services for special populations in «t^he following areas: 
quality of instructional offerings, guidance and 4 counseling, 
placement and follow-up services, capacity and condition of 4 c . 
facilities , ^equipment used, and other services. ^Further, Colorado 
"failed to perform process evaluation across all special population 
subgroups. * ■ ; \ ' > \ y ' • - • 



t 



Measuring Program Outcomes, 



The MIS, V through its Student Accountability Data Tool, collected 
information on all categories of special population completers' and 
leavers (USOE 6-digit code) 'at the institutional level However, the 
following limitations in meeting VEDS requirements were found: 
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a. > Although VE-135 collected data on all spetiM population 
v subgroups, it did not publish follow-up diiPa on the 

different ,subgrou|>s. 

b. 6 ^ Its follow-up study d4^d not include employer 
* ^satisfaction. . ' * 

At the time of the study , Colorado also did* not have > the ability 
to measure student , performance or achievement. «, 

. w Connecticut ' 

Deitgrmining Program Accessibilty %< 0 : * 

The Vocational* Evaluation System of Connecticut determined 
program accessibility for * special ^populations through the . 
Self-Evaluation Instrument. Data gathered on accessibility 
in&^uded the following: \: 

a. Whether «sex stereotyping was eliminated in program* 
publicity * 

•b. Whether publicity was directed to "encourage minority and 
handicapped students to enroll in all vocational areas" 

' ^ / . .. ^ , . / * 

; x - ■■ . , ,« * v 
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c. Whether facilities were easily assessible to the 

handicapped ^ \ " "* 

' ' ■ • M: ' ' ' . 

„ ■ * . <~ 

d. Whether there was! an "afction plan in effect to overcome sex 
stereotyping in recruitment" ' 

\ ' C ■ ■ r s 

e. Whether vocation^ education programs* were, 

- comprehensive »andfl|not limited to single courses 

The data collected were limited to* certain special population 
subgrfoups. Items a, b, axi& d s,hould have been directed to all ■ > 
special population subgroups. "Further, evaluation o^iocal school 
efforts to remove attitud|jpal and societal barriers fw.cts not 
included. * S * / 

• . • ■ . if. • . - ■ . .. 

Determining ParyicipatiQji \[ '" T ■ / ^ 



The series" of research reports gn enrollment issued by the 
Bureau of Vocational Program Planning and Development satisfied all 
of the VEDS requirements e;&cept the following: 

a. Handicapped enrollment by instructional^ settings , a . 

b. Categories of disadvantaged enrollment ' ; , • 

c. Categories of handicapped enrollment 




d. Limited English-proficiency enrollment 

On the other hand, the prj^&e^. staff was informed by the bureau 
that the Department of J Educ^^^^^t^' System collected data by 
handicapping conditio^, se ^fs^5^' predominating language. Tfre 
department could determine ^^^^m^^K^\^a^i.<y^ of handicapped in 
vocational programs i\n th^f SQg^§^^digit and two-digit codes. 

Identifying Services 

Proems evaluation was accomplished by using a self-eivaluation 
instrument. These were thev major limitations with regard to the 
identification services for special populations: . 1 m 

' 0 , I ' ^ ' ' ■ • 

a. Concern for facilities and -equipment was limited to the 
issue of accessibility for the handicapped. . . 




b. , There were no specific question regard fhg the determination 

of quality of -counseling, placement, and follow-up* services 

for special populations. - . • 

■ * '* , * * 

■ i * & 

c. Concern for curriculum was limited to the review of < • 
.* , curriculum materials to' overcome, sex bias. 

Measuring- Program Outcomes * ■* ~* ? - 

Data and reports on atfeent follow-up were collected- and 
published by the Pureau of Vocational Program Planning and 
Development. Tt^e most recent published reports (.1978-79) showe/3 
that the bureau had the capacity and ability to meet all VEDS 
requirements except the following,:, 

1. Foll&W-up of vocational graduates'by racial/ethnic, sex 

and handicapped '° • ^ , 

2. ^Determination Of employer' satisfaction with . special 
; *. /• » population vocational completers and leavers 

* ** * ' • ,. 

.At the time of the study, the'state £id not have a program to 
measure student achievement. V^A v # 




CHAPTER VII 



TECHNICAL ASSISTANCE PROVIDED 
TO STATES 

t 



This chapter presents a summary of the major strengths and 
weaknesses in the ability of the four cooperating states to evaluate 
programs and services for special populations;, the recommendations 
made by the project/staf f , and the action and decisions made by the 
states in response to these recommendations, This chapter also 
discusses the additional technical assistance provided to help the N 
states implement chosen solutions to identified problems. An 
end-of-the-year profile is presented to show the progress each state 
has achieved during the year. 



Alabama 



Summary and Recommendations 

Examination of the instruments used and information available in 
Alabama regarding the evaluation of programs and services for 
special populations reveals a N number of limitations, especially in 
its ability to generate data for compliance and planning 
requirements-. These are summarized as follows: 

1, Alabama's ability to determine program accessibility was ^ 
generally limited to data regarding equal access for male 
and female students as well as determination of physical 
barriers for the handicapped. The state was not aware of 
recruitment procedures employed at the local school level, 
the .local entrance requirements, and the state's own 
progress in removing atbdtudinal and societal barriers. In 
view of the foregoing, the state fell short in its ability 
to comply with federal regulations thatrequire states to 
report periodically to the Office of Civil Rights on the 
accessibility of their vdcational programs, 

\ Recommendations Made : The items in the Instructional 
Program Review Instrument, interview outlines, and 
Standard of Policies should be reexamined in relation 
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to the requirements of the Education Amendments of 
*■ > 1976 and the Civil Rights 'Act of 1964, These 

instruments and documents should be modified to \ 
incorporate not 'only legislative requirements but 
also data needed for planning. Alabama should gather 
data on program accessibility across ail special 
population subgroups and -prepare a report on the 
subject for tile* Office of Civil Rights, 

Through the MIS, Alabama haj3 the ability to generate data on 
enrollment for the different special populations categories 
of as required by -^EDS except for enrollment of handicapped 
instructional settings and enrollment of persons wi$h 
limited English proficiency (LEP). 

¥ 

% Recommendations Made : Alabama needs to modify the MIS 
4 .enrollment and follow-up forms to incorporate VBDS 



) requirements, ^ 



The ability of the state to perform process evaluation ofi 
programs and services -for special populations was/ limited 
to instructional fairness in terms of gender and 
determination of additional services for handicapped, > 
disadvantaged, and women to the exclusion of LEP. Alabama's 
vocational edugatidn evaluation system did not investigate 
the following procesg elements: quality of instructional 
offerings for specif populations , guidance and counseling 
for special populations, placement and follow-up services 
for special populations, capacity and condition of 
facilities and equipment for special populations, and other 
services for special populations. In view of these 
limitations, Alabama was. unable to provide the information 
required by the Education Amendments of 1976. 

# 

Recommendations Made : The Instructional Program 
Review Instrument (IPRI) should be modified to enable 
the state to perform process evaluation across all 
,f ' special population subgroups. Further, special 

attention should be given to those process elements 
which were not included in the IPRI as noted in 3. 

Alabama did not have the ability to determine program - 
outcomes as required by the Educa^ipn Amendment of 
1976 which mandated that states measure student 
achievement ulsing standard proficiency measures such 
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as criterion-referenced tests* Furthermore , the 
follow-up system did not meet VEDS data requirements. 

Recommendations Made ; Plans should be developed , 
tested , and implemented on the determination of 
T student outcomes , especially the mandated federal s 
requirements. 'It is also suggested that Alabama \ 
prepare a separate annual report on the 
evaluation of special population enrollment' in 
vocational education reflecting federal * 
legislative and planning requirements. Further, 
an annual report should be prepared and submitted 
to' the Office of Civil Rights to substantiate the 
state's effort to elimi?fate discriminatory 4 
practices ^jpn vocational education. . » • 

General Recommendations Made * 

In addition to evaluation requirements mandated by federal 
legislation ,. there are some recommendations which relate to 
the, total evaluation system which should be implemented if 
maximum impact and efficacy is to be achieved by the system 
for both regular and special students . 1 

1. The total evaluation procedure should b^ formed 

into a unified system. This does not necessarily mean 
reorganization. It does mean that clear lines of 
responsibility and authority should be estabished 
within the system. The ^ata needed r the time, and form 
in which these will be provided; the relationship 
between the area specialist review and the team review; 
the use which is to be made of the, findings are, all 
matters which shbuld be carefully and clearly defined. 
It is essential that the individual responsible for 
directing evaluation activities have* the authority to 
operate the system effectively. 

2. It is further recommended that the total process of 
needs assessment, planning, and evaluation be * 
described in detail. This will assist the state in 
establishing the relationship between quality, 
dollars, and future planning. , 

3. An evaluation system for postsecdndary vocational 
education should be designed, tested, and 
imEfSfemented . At the^tiine of this study, there 



1«1" * , % 
103 



v was no well-defined vocational evaluation system 

A for postsecondary vocational education. 

4. The goal of assuring vocational education to 
any citizen who desires it 'should be 

accepted by the state, and a well-defined plan for 
making occupational training accessible to all 
, should be designed. / .'....» 

5. Comparative data about programs needs to be gathered 
and provided to schools and evaluation teams* 
Information on enrollment (regular and special 
stud'ents) and follow-up along with other data 
elements, such as dollars invested, staff education 
and experience, facilities and equipment, and 
quality should be available on each program to be 

„ evaluated. If this data can be presented in a 1 ^ 
comparable vinode artd with similar program averages it 
will be more meaningful and usable. „ - . 

• * , - 

j 

Technical Assistance Provided <and Results * 
first Quarter (February and March only) 

o Finalized memorandum of understanding between Alabama and the 
National Center regarding technical assistance on evaluation 
for Year II . 

Second Quarter J 

t o Analyzed Alabama' s ability to evaluate programs and 
services for special populations. Activities included 

r - collecting and analyzing pertinent data and 

documents from the state; 

interviewing state officials concerned with the 
evaluation of special populations enrolled in vocational 
programs. 

o. Preliminary analysis report was made and sent to^ Alabama 
for reaction. The state was also furnished copies of the 
"Evaluation Matrix" and the summary of pertinent legislation 
developed by the technical assistance staff. 

o In the second week of June, a visit was made to Alabama to 
discuss the 'basis/criteria of the analysis study with 
emphasis on legislative requirements, the results of the ^ 
analysis, and the state's reaction. 
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Third Quarter ~ v 

• — ? 

*o\ The report was finalized. This included clarifying tlibse 

parts on question and making specific recommendations v 
to solve identified problems. 

o. On the third week of July, a visit ^was made to discuss 
/Specific recommendations for improvement and to help the 
state get started in terms of implementing the '• 
recommendations. Specific weaknesses of thQ "Standards and 
Policies 11 and t;he evaluation instrument were discussed by the 
consultant with the Research and Coordinating Unit (RCUU 
staff/ Suggested solutions/strategies were also'made. The RCU 
staff promised to study further mandated evaluation 
requirements and the "Evaluation Matrix." They agreed to 
x revise their "Standards and Policies" and cJevelop hew 
evaluation instruments that would reflect legislative ' 
requirements and the elements of an effective evaluation 
system. ■ ^ « - • 

Fourth Quarter ^ 1 

o In the third week of October, a visit was made to review 
accomplishments and heljjr resolve some major problems. 

o Evaluation instruments were revised based on recommendations 
of the project staff. 

o "Standards and Policies for Quality Programs in Secondary 

Schools" was revised to include both planning and legislative 
requirements. This will be submitted to the Alabama State 
Board for Vocational Education this spring, for approval. 

o Report on enrollment and> program completion forms were 
revised to include all the VEIjfe data requirements. ^ 

o Plan for longitudinal follow-up study of . special population 
program completers and leavfers is" being developed. 

o State-wide testing of student achievement is under study. 
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Information Co 11 ere ted 



Start of Technical Assistance 
Aprir 1979 ^ 



Actions Taken 

April - December 1979 



< Status 
December 30, 1979 



1. Program Accessibility 
1.1 Facility factors 

w 

' 1.2 Educational factors 

r 

* 1.3 „ Societal factors 



- Limited to physical barriers 
for the handicapped 

~ Limited to sex designations 

- None 



Instrument revised to include 
all facility factors 
Instrument revised to include 
all educational factors 
Annual application reg. revised 



Complete 
Complete 
omplete 



2. Participation 



2.1 



Enrollment (VEDS 
classification) USOE 
\^ 2-digit code 

2.2 Enrollment {VEDS 

classification) USOE 
6-digit gode 



Complete except LEP and^ 
instructional settings 



Complete except LEP and 
instructional settings 



Enrollment forms were 
revised to include 
all VEDS requirements 



- Complete 



- Complete 



H 

o 



3* Proces^ 

3.1 Quality of instructional 
offerings 

\ 

3.2 Capacity condition of 
facilities & equipment 

3.3 Guidance % counseling 

3.4 Placement & follow-up services 



- ^one 

- None 

- None 

- None 



Evaluation instruments 
were revised to enable 
tW& state to evaluate 
3.1 to 3.4. 



Complete 

Complete 

Complete 
Complete 





- Under study 

- Complete 

- Under study 

- Under study 



4. Outcomes 

4.1 Student Achievement - None 

4.2 Successful program completers - "Limited 

4.3 Succesful placement - Limited 
4*4 Successful employment over time - None 



- Under study % 

- Report of program completion 
forms were revised ^ 

- Longitudinal follow-up study of 
special populations is being 
planned to meet 4.3 & 4.4 



Figure 12. An end-of ^the-year profile of the Alabama, secondary vocations^ education 
evaluation system. 
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California 

Secondary . The California Secondary Vocational Evaluation System 
with its PAR, PAVE, FUSE, - an& MIS components had the fo-llowinlj 
strengths and weaknesses with regard to its ability to evaluate 
programs and services for special populations: # 

!• California-^ ability to determine program accessibility 
was limitea to documentation of program barriers for 
the handicapped and the disadvantaged, efforts to .! 
encourage special populations to enroll including 
prerequisites for enrollment, 'and equal access to 
programs for both male and female students. 

*' , ' 

Recommendations Made ; Program accessibility 
\ * should be determined across all special ^ 

population subgroups. This will require .""""••* 
modifications in the PAVE instruments. 
Further , . a decision' regarding ^lich 
instrument' to use to collect data |howing ^~ 
local school efforts gftOi remove attitujtehal 
and societal barja^fers needs to be mades An 
• ^ annual report on^program accessibility 

highlighting the state's effort to remove" 
various discriminatory 'practices should be . 
prepared and submitted to the Office of * 
0 ■ * Civil Rights. , 

. -■ 2. California had the ability to determine, participation V 

according to sex /designation and ethnic classification - 
at both the USOE six-digit and two-digit codes., \ 
^However, the state did not have the ability to meet the 
following VEDS data requirements: * enrollment as to | 
categories of handicapped and f£heir instructional 
settings and ^categories of disadvantaged. Further, the 
state did not collect data about limited English 
- proficiency students either at the USOE two-digit and 
six-digit codes. ; ^ ( 

Recommendations Made : The MIS-VEA Form 48 should 
be modified to make it possible for California to 
gather all the data required by VEDS. 

3. California's ability to perform process evaluation 
focused mainly on the disadvantaged and the 
hahdicappped . 



f 



Recommendations *Made : Process evaluation should 
be expanded to all special population subgroups 
in order to include women* minorities, and limited 
^ English proficiency students., To accomplish 
■ . * this, ife is suggested that the PAVE instruments be 

' modified. * 

4. California^ ability to measure outcomes did not meet 

She requirements of VEDS and the Education Amendments . 
f 1976 although the state was considering some plans 
to this effect. v 

_^ % ' Recommendation^ Made : The requirments of VEDS and 

^ the Education Amendment of 1976 in terms of / 

measuring student achievement stooald be carefully 
studied and" incorporated into the plans being 
p * developed*. Due consideration to the 'state 1 s 

prevailing fiscal and social constraints sould b6 
given. 

Technical Assistance Provided and Results 4 - • * - 
First Quarter (February arid March only)^ 

O Finalized memorandum of understanding 'between the California 
Vocational Evaluation Division and the National Center <, 
regarding the technical assistance project on Year II. 

¥ Second Quarter * 

— ', * ' * . 

o* Analyzed the state's ability to evaluate programs and 

services for special populations. Activities included 

- collecting and analyzing pertinent data and documents - 
from the state; *■ , - \ >• - 

interviewing state officials concerned with the 
evaluation of special populations enrolled in 
vocational programs. ■ ' % 

o frepared, preliminary report and sent tof the state in the 
second week of " May. The state was also furnisl§ed with 
copies' of the "Evaluation Matrix 11 and 6 the summary of 
pertinent legislation developed by the technical 
assistance staff. : . . . 

o State replied asking further clarifications on certain 
aspects of the report* 
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report was prepared which Included clarifiying those, parts' 
question and makincp specific recommendations tcA solve 
identified problems. 

* * * ' > s : 

o '\ visit wars made \yf the project director in the second 
week of July to discuss with the secondary .vocational 
education evaluation officials /the basis/criteria of the 
analysis study with emphasis on legislative requirements , 
results of the analysis, and specific recommendations to 
deal with the identified problems. 

o On August 1, 1979 the secondary vocational evaluation 
consultant prepared a memo to the California Vocational 
Education Executive Staff (VEES) concerning the result of 
• the national Centers tudy. He also informed the VEES of % . 
the specific alternatives he thought were most .appropriate 
in solving identified problems. Previous -to* this memo, 
however , the evaluation consultant discussed the analysis 
report with his own staff and field personnel concerned 
with vocational evaluation. Thteir opinions formed the basis 
of a portion of £he consultants memo. 

Fourth Quarter ^ ' 

o A visit was made by the project director during the 

first week of October to attend the Follow-up of Student 
and Employers (FUSE) advisory meeting^ and assess the * - 
% accomplishments of California with the evaluation 
consultant. 

o Evaluation of the PAVE and PAR instruments were revised on v 
the basis of recommendations of the technical assistance 
staff regarding determination of program accessibility. 
California's VEES approved all the recommendations of the 
secondary vocational education consultant. -Among efforts 
to implement approved recommendations, a state financed 
and supervised tfesting service (for districts wishing to 4 , 
participate) was started to measure special^ student 
achievement. 
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Information Collected 


Start of Technical Assistance 
April 197* 


Actions Taken 

April - December 1979 


Status 
December 30, 1979 


' 1# , Program Accessibility N 








1*1 Facility factors 
. . 1.2 Educational factors 


-Limited 
- Limited 

• 


Revised evaluation 

\ nofrnTnpnha fn i np 1 11/^0 

1.3 to 1.3. 


- Complete 

- Complete 


1.3 Societal factors 


- Limited ^ 




<5 Complete 


2. Participation 


• * • - s . 






2-1 Enrollment (VEDS 

classification) USOE 
2-digit code 

" 2.2 Enrollment (VEDS 

- classification) USOE 
6-digit code 


- Limited to sex designation 
" and minorities 

♦ r * 

— Limited to sex designation 
and minorities ^ 


Suggestions & recommenda- 
tions of the technical 
assistance staff were studied 
by the vocational education 
personnel and .brought to the 
attention of the VEES 
- \ 


Under Study ^ 

> 



o 

00 



3. Process 

r 

3.1 Quality of instructional 
offerings ^ 

3.2 Capacity condition of x 
facilities & equipment 

3.3 Guidance & counseling ^ 

3.4 Placement & follow-up services 



Limited to disadvantaged 
and handicapped 

- do - 



- .do - 

- do - 



Suggestions & recommenda w 
tions of thei technical 
assistance sraff were studied 
by vocational, education 
personnel & brought to the 
attention of the VEES 



Under study * 
for possible 
incorporation 
in the exist- 
ing evalua- 
tion instru- 
ments 



4. Outcomes 

4.1 Student Achievement - None 

4.2 Successful program completers - None 

4.3 Succesful placement - None 

4.4 Successful employment over time -None 



VEES approved measurement of 
special population student * 
achievement for districts 
wishing to participate 

FUSE) system being developed 
to -meet 4.2 to 4.4. 



Instrumenta- 
tion being 
developed 

Instrument ac- 
tion being 
developed 
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7 Figure 13. An end-of-the-year , prof ile of the 
% educatipn evaluation system, 



California secondary vocational 
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Pbs tsecondary . The Caldxornia Pos tsecondary Vocational 
Evaluation System with its C0PES and SAM w^5 determined - to have the 
following strengths and weaknesses: 

1. The ability of COPES to determine program accessibility 
focused pn the issue of sex equity and* architectural « 
barriers for the handicapped. * * ■ m , \~ 

Recommendations Made : The coverage of CQPES 
instruments should be expanded in order examine 
program accessibility across all special 
population }Su%[roups (Women, limited English 
Speaking, minority, disadvantaged, and 
handicapped ) , in terms of facility, educational, « 
and societal factors. It is further suggested 
that PAR address program accessibility, especially 
f local school ^forts to remove attitudinal and 

societal barri|rg. 

■ * • ■ « 

2., SAM has the ability* to determine participation of % 
special populatiorite^not only for compliance but alsc? 
for jprogram improvement. However, enrollment of the 
various categories Qf special populations was ^ 
determined atV the; 3QSDE two-digit codes only in viW 
of the unresolved difficulty associated With the j 
reporting of pos tsecondary enrollment at the USOE 
' six-digit code. f 

Re comme nd a t iphs • Made : SAM needs to demonstrate 
' " its ability t<> pollect and disseminate data 

conce^in^4v^pd,lcapped enrollment by instructional 
settings, cat?egt>ries of disadvantaged and 
* handicapped enrollment, and limited 

■p - English-speakiri^ enrollment by USOE six-digit 

, V jcode. 

3. COPES has thje ^abili^r to identify additional 

services for all special populations. However, the 
technical assistance ^taff had not been able to 
evaluate any C01ES report on the subject. 

Recommendations .tifode: COPES mus-t report specific 
process evaluation of programs and services for 
special* populations. In addition it should also 
inclj^ae response^iof special population students. 
This jpiay mean alluring the COPES student 
respondent sampling plan. It must include a 
definite proportion of the major special 
population subgs%ips and their respective 
responses mus-t b$|;separately treated in the COPES 
^ reports. 
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4. SAM does not have the ability to measure, program 
outcomes including a longitudinal study of 
vocational prpgram completers and leavers* . 
4 Further^<tha state did hot hav£ any system for • 

testin^^fcudent achievement in postsecondary 
vocatioiial program^. - " ^ 

\ ■ ' > - ;' . • ~ * 

*■• * Recommendations Made : ' A system of determining 
^ . % employer satisfaction and measuring student * 

Achievement for both regular students and special 
populations must b§ planned, .tested, and 
implemented if the California Postsecondary^ 
Vocationai^Evaluation system is to comply with 
the • Educati^t- Amendments of 1976. ^jf 

; ' , * J ■ ' ; .. ' / , V • *: m ■ . 

Technical Assistance Pr &vided and R e^uyts . 

■ : • v . , j] ■>,■;■•■ ; v. 

First Quarter ( February, and MarcWonly ) x * 

q Finalized memor and uwKof understanding between the National 
' Center and the OfficeVof the Chancellors regarding the & ^ 
technical assistance ^r^ect "for Year II.* * 

Second Quarter ^ 

. * .-. * , ,• ' ' ■ ■ 

o* Analysis of the v state's ability to fe^aluatie. programs 
. and services for speelal pqjtuiat ions was started in 

April. Activities iricluded: " • 1 % - m ■ 



- collecting and analysing pertinent data and 
documents from "the ^tafce; r ^ . * 

» -V * 

- interviewing state officials concerned with t|ie 
evaluation of special* populations enrolled in vocatit^al 
programs. • ^ 

o Preliminary report was made ;dnd 'ient to the state * 

during the second week ib May,/ The state i was also, f urnishea ' 
with copies of the "Evaluation* Matrix" &nd the summary 
of pertinent legislation developed by the technical * \ ? 
assistance staff. T 

o No written reply Was received from the state during this 
period. av 
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Third Quarter 



o A visit was made during the second week of July to 
discuss with the vocational evaluation specialist of 
the postsecondary. system and some of his staff the 
basis/criteria of the analysis study with emphasis on 
requirements and results of the analysis. Instead of - 
reacting directly to the total report, the specialist 
requested more information on how the postsecondary 
system could meet VEDS requirements • in this regard/ 
a detailed discussion was made on the VEDS requirement 
and the limitation of the postsecondary system. 

o California finalized its plan for a longitudinal 
follow-mp study of .vocational education students 
including special populations. .Project director was 
requested to serve as a member of the advisory committee. 

Fourth Quarter - 

■ i 

o Visited California in the third week of October and 
conferred with the postsecondary evaluation consultant 
regarding the progress of the state in implementing the 
rectrtnmendations of the technical assistance staff* 

o Visited the state in the first week of November to attend a 
meeting on SAM 1 s longitudinal follow-up study of vocational 
education program completers and leavers. 



Colorado 



Summary and Recommendations 

The Colorado vocational evaluation system with its Supervisory 
Checklist, Comprehensive Program Review, and MIS had the following 
strengths and limitations with regard to its ability to evaluate 
programs and services for special populations* 

1. Information collected on program accessibility was 
primarily concerned with the determination of 
physical barriers for the handicapped and sex 
equity in recruitment and selection of students. 

Reconmiendations Made : Determination of program 
accessibility needs to be expanded to include all 
special population subgroups in such areas as facility 
factors, educational factors, and societal factors. 
Decisions must be made with regard to* the specific data 
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Information Collected 



Start of Technical Assistance 
April 1979 



Action Taken 
April - December 1979 



Status 
December 30, 1979 



1, Program Accessibility 

1.1 Facility factors 

1.2 Educational factors 

1.3 Societal factors 



- Limited to architectural 
barriers 

- Limited to sex equity 

f - None 



Suggestions & recommenda- 
tions were being 
studied 



- Same 

- Same 

- Same 



2. Participation 

2.1 Enrollment {VEDS 

classification) USOE 
2-digit code 

2.2 x Enrollment (VEDS 

x classification) USOE 
6-digit code 



Complete except instructional 
settings 



- None 



- Being studied 



Asked further clarifications 
from VEDS due to technical 
difficulties 



- Hill initiate 



- Status quo 



H 
to 



Process 

3.1 Quality of instructional - Limited 
offerings 

3.2 Capacity condition of - Limited 
facilities ft equipment 

3.3 Guidance & counseling - Limited 

3.4 Placement & follow-up services - Limited 



v 



Suggestions ft recommenda- 
tions were being 
studied 



Plans to ini~ 
tiate process 
of items 3.1 
to 3.4 



4. Outcomes 

4.1 Student Achievement - None 

4.2 Successful program completers - Limited 

4.3 Succesful placement t - Limited 



4.4 Successful employment over time - None 



- No action 

- Planning development of 
a longitudinal follow-up 
study of program 
completers and leavers 



- Same 

- Testing of . 
instruments 
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Figure 14. An end-of-the-year profile of the California postsecondary vocational 
education evaluation system. 
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which should be collected by the Comprehensive Program 
Review (CPR), supervisory checklist, and institutional 
application plan. It is important that the state get a 
complete picture of program accessibility. 

Data collected on participation of special populations 
satisfied VEDS requirements except handicapped 
enrollment by instructional settings. 

Recommendations Made : MIS forms must be modified to 
include data on handicapped enrollment 
instructional settings. 

Process evaluation was targeted principally for regular 
students. 

Recommendations Made : The Education Amendments of 1976 
require process evaluation of programs and services for 
all special' population categories. This ^means that a 
state should be able to compare or contrast regular 
students and special populations in terms* of qualify and 
availability of instructional of ferings; guidance , 
counseling, placement, and follbw-up services; and y 
capacity dnd condition of facilities and equipment. 
Modification of the CPR instruments and supervisory 
checklists are, therefore, recommended in this regard. 
-■*••' 

Data collected on vocational program. completers and leavers 
satisfied VEDS' data requirements except that for 
employment over a period o£ time. Further, the state did not 
have the ability to measure student achievement with 
Standard prof iciency measures such as criterioh-referenced 
tests. : 

Recommendations Made : The follow-up system Of the 
state must be upgraded to include a longitudinal 
study of special population program completers and 
leavers. Further, necessary modifications need to 
be made in the computer programming to enable the 
state to compare regular and special populations. 
The state also needs to plan for a 4 state initiated 
> testing of student achievement if it is to comply 
With the Education Amendments of 1976; t 
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Technical Assistance Provided and Results 



First Quarter (February and March only) 

o Finalized memorandum of understanding with Colorado regarding 
Technical Assistance in Evaluation for Year. II. 

Second Quarter * - ; . . 

o Analyzed the state's ability to evaluate programs and jg 
services for special populations • Activities included < 

- collecting and analyzing pertinent data and documents 
from the state? 

interviewing state personnel concerned with the 
evaluation of special populations enrolled in vocational 
programs. 

o Preliminary analysis report was made and sent to the state 
for reaction. The^taf&swas also furnished, copies of the 
"Evaluation Matriar* and of pertinent legislation summary . 4 . 
developed by the technical st^ff. 

Third Quarter 



A visit was made by the project director during the second 
week of July to discuss with state officials the basis of the 
V"\ analysis report, mandated evaluation requirements/ evaluation 
V vmatrix, and the results of the study. 

o Colorado responded with favorable written comments 
b concerning the report • 

^ o A final report and irecommendations were made. The state was 

furbished a copy. Reaction of the state to the 
fecbmmendations was also favorable. 

o Colorado asked specific assistance in the 
*> modification/revision of 'evaluation instruments that were 

1 used. 

o A detailed critique including ^specif ic suggestions for 

revision on the supervisory checklists, CPBT instrument, and 
' _ • institutional application plan was made and sent. 
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information Collected 



1 Start of Technical Assistance 
April 1979 



Actions Taken 
April - December 1979 



Status 
December 30 , 1979 



1, Program Accessibility 
1.1 Facility factors 



1.2 Educational factors 

1.3 Societal factors 



- Limited to physical barriers 
for the handicapped 

- None 

- None * 



Suggestions & recommenda- 
tion% were studied and 
revision of instruments were 
started 



Being tested 
Being tested 
Nbne : 



2. Participation ' 

2.1 Enrollment (VEDS 

' classification) USOE 
2-drgit code 

2.2, Enrollment (VEDS 

classification) USOE 
6-digi£ code 



- Complete except .instruc- 
tional settings 



- do- 



Complied with all the 
VEDS data requirements 



Complete 
Complete 



3. Process 


/ 








3.1 
* 3.2 


Quality of instructional 
offerings 

Capacity condition of 
facilities & equipment 




Items 3.1 to 3.4 were, 
limited to regular students 
4n mixed class settings 

» ■* 


Suggetions tf recommendations 
were being studied and 
revision of evaluation 
instruments were started 


Evaluation 
instruments are 
being 
tested 

* . * 


3.3 


Guidance & counseling 










3.4 


Placement & follow-up services 
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4. Outcomes 

4.1 Student /Achievement - None • 

4.2 Successful program completers - Complete 

4.3 Successful placement -.Complete 

4.4 Successful employment over time - None 



- Recommendations studied 



- Recommendations studied 



- Status quo 

- Complete 

- Complete 

- Planning stage 




Figured 15. 
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An ^nd-of-the-year profile of the Colorado vocational education evaluation 
system. Xyi 

. » ■ . ' * .. 



Fourth Quarter ^ •" 

o During thisj^eriod, Colorado reorganized the Occupation 

Education mvisionv This included redefinition of roj^s and 
responsibilities of t;he branchesoand sections under €he 
division. Further, some personnel were transferred. In view 
of the foregoing developments r the state vocational education 
evaluation personnel made only few significant 
accomplishments. Suggestions a tfd recommend a tions^fcor 
improvement were studied^and revisions" of "the evaluation 
instruments and program application guidelines were started J 



r 

Connecticut 



Summary and Recommendations 



The Connecticut Vocational Evaluation system with its 
Self-Evaluation and MIS components had the following strengths and 
weaknesses in its abiliy to evaluate programs and services for 
special populations: 

1. Data collected on program accessibility were limited to 
certain, special population subgroups in such items as 
program publicity;' recruitment, and diversity of 
program offerings. 

Recommendations Made : There is a need to expand the 
coverage of the self-evaluation instrument in the # 
determination of program accessibility. Program 
accessibility must be examined across all special 
population subgroups in terms of facility/ 
educational, and societal factors. 

* 2. Connecticut enrollment data satisfied the VEDS requirements 
except for the following: handicapped enrollment by 
instructional settings, categories of disadvantaged 
enrollment, and limited English proficiency enrollment. 

Recommendations Made : The ability of Connecticut to 
determine participation ofb special populations 
enrolled in vocational programs must be upgraded, 
r Necessary modifications must be made in the 
enrollment forms and computer programming to 
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enable the state to collect and disseminate data 
regarding handicapped enrollment by instructional 
settings , categories of disadvantaged enrollment, 
arid limited English proficiency enrollment 
by USOE six-digit and two-digit "codes. \ 

The ability of the Department of Education Data 
System to identify special populations enrolled in 
k, vocational education was limited to certain subgroups. 
It is , therefore, recommended that the Division of 
Vocational Education continue to negotiate with the 
Department of Education in terms of the identification 
of all special population .subgroups in vocational 
education ptograms. The present mechanism may be most . 
advantageous in terms of cost. At the same (time, it 
enables the Division of Vocational Education to see the 
extent to which its programs serve the total special 
populations who are in school. * . . 
. v " ■ ' 

EProcess evaluation was limited to the examination of 
facilities and equipment for the handicapped and the 
review of curriculum materials for^the handicapped. 

Recommendations Made : There" is a n€>ed to> restructure 
the s^lf-evaluation instrument and to make 
methodological modifications to enable the state td 
compare and Contrast special populations and regular 
students in terms* of quality of instructional 
offering^; guidance; counseling , placement, and 
follow-up services; and capacity and condition of 
facilities ancf equipment. 

Jhe state has the capacity ^nd ability to meet VEDS data 
requirements except for the follow-up of vocational graduate 
by racial/ethnic, sex, and handicapped designations, a^d 
determination of ^Ploye* satisfaction. At the- time, of . 
th^e study, the state did not have the ability to 
measures tuderit achievement. * 

Recommendations Made : Necessary modifications must be 
made in the follow-up forms being used to enable the 
state to collect data pn handicapped vocational 
graduates as required by VEDS. Furthermore, a system 
of determining employer satisfaction, successful 
employment over time, and measuring* student * 
achievement must be planned, tested, and implemented 
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if Connecticut is to comply with the mandate of the 
- , Education Amendments of 1976 •» 

Technical Assistance Provided and Results y ■ 

* First Quartfer . * 

o No activity. ^Connecticut was, not yet selected as a 
cooperating state* * 

Second Quarter 

— — ' . * 

o Finalized memorandum of understanding between the National 
Cefiter and Connecticut for the Technical Assistance ^Project 
on Year II. 

o Oriented s'tate vocational evaluation s^taff regarding 
* objectives of the project. ° 



o Set preliminary tiysa^tables for the technical assistance 
activities with the state* v 



Third Quarter 



< • 



6 Analyzed the state's ability to evaluate programs and 
services for special populations. Activities included 

- collecting and analyzing pertinent data and documents 
from the state; 

interviewing state officials concerned with evaluation 
f of programs and services for ^special populations. 



^9 



Preliminary analysis report was prepared andPsent to the 
state . 

State was also furnished with the "Evaluation Matrix" and t£e 
summary of pertinent laws on evaluaffioTK of vocational 



programs . 
Fourth Quarter 



p Final analysis report and recommendations were sent to the 
state • * * 

o Project director v^sitecl the state to follow-up 

recommendations and plan with the state vocational education- 
evaluation personnel future technical assistance^ 
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. * 


* 

f " • 


* 


s . ■ • 


information collected 


. Start of Tecnnlcal Assistance 
^ April 1979 


Actions Taken 

April - December 1979 


Status 
December 3K), 1979 


1. ' Program Accessibility* 

1.1 traciiity £ actors 

1.2 Educational factors 


> \— iii mi tea to arcnitectuai 

barriers for the handicapped 
- Limited to recruitment 


no sign it leant action 
was taken 


— Status quo 

- Status quo 


1.3 Societal factors 


- None 




- Status quo 



2. 



Participation 
2.1 



2.2 



Enrollment (VEDS 
classification) USOE 
2-digit code 

Enrollment (VEDS 
classification) USOE 
6-digit code 



Limited to sex designation, 
minorities, ft ^handicapped 

- do - 



Suggestions ft recommendations 
were studied. Explored with 
the Education Department re- 
garding the possibility of * 
using the latter'.s data to 
comply with VEDS data req. 



Negotiations * 
with Education 
Department is 
continuing 



H 
O 



Process 

3.1 Quality of instructional 
offerings 

3.2 Capacity condition of 
facilities ft equipment 

3.3 Guidance ft counseling 

3.4 Placement ft follow-up services 



Limited to review of mate- 
rials to overcome -sex bias' 

Limited to accessibility for 
the handicapped 

None ■ 

None . 



No signif icant_action was 
taken 



- Status quo 
-. Status quo 

- Status quo 

- Status 1 quo 



4. Outcomes 

4.1 Student Achievement 
* 4.2 Successful program completers 

4.3 Succesfu! placem/mt 

4.4 Successful employment over time - None 



- None 

- Limited to sex designation, 
minorities 

- do - 



- No action 

Suggestions ft recommendations 
were studied 



Status quo 
Under planning 
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Figure 16. 



13 i 



An end-of-the-year^profi-le of the Connecticut vocational education 
evaluation system. p 
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CHAPTER VIII . |l 



GENERALIZATIONS AND REGCjjlMEjjpATIONS 

" - V.O ' ' V • ' : 7 • • ■ 

r In i. 

- ' f\: * " » ■ • ' 

, Thirs chapter presents the major generalizations, regarding 
the four assisted states 1 abilities to evaluate programs and 
services for special populations enrolled m vocational 
education programs. In addition, general r^commend^ations are 
also made for other states that may encounter similar problems 

Generalizations 

■•' f ■ ' — r — ■ v.'. • 

1, Determination of program accessibility is a common v \ 
problem among the four assisted is tateS^. Program 
accessibility is generally determined in terms of sex 
designations and handicapping conditions . Further, 
states have problems complying with the requirements 
of the Office of Civil Rights, . 

2, Determining participation of special populations 
enrolled in vocational programs through compliance 
with the VEDS data requirements poses /little problem 
at the secondary level. The four, assisted^ states are 
at different stages of implementation of an 
information system, with Colotado in the most advanced 
stage of development. However , . at the postsecondary 

' level, compliance with the VEDS data requirements is 

still frajught with technical difficulties as 
demonstrated by the California case, \ - 

3, Identifying additional services for special \ 
populations is a comftiom problem among the four 
assisted states , information' on additional 
services is generally limited to the handicapped 
and women to the exclusion of limited English 
proficiency (LEP), minority, and disadvantaged 

* individuals. Further, such areas as guidance, 
counseling, and placement for special * 
v populations are Mot generally included in the - 

evaluation, / . 

' ' j ' ■ * s . . , 

4, Measuring student outcomes is a common problem amcjjng # - 
the four assisted states. This is especially critical 
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in terms of complying with tne Education Amendments of 
* 1976 which mandated that states measure student 

achievement vfith standard prbficiency examination^/ 
' Further, states have a common problem of conducting? 

, longitudinal f olloW-u£ studies , although California is 
starting to develop such a fpllow-up ate both secondary 
and postsecondary levels.- 

5. The limitations , some of which, are very serious, in 
the abilities of the four assisted states to evaluate 
' programs and services for special populations can be 
attributed to the following: 

* i- a. Failure of the states tp identify the key 
elements of ah; evaluation system whitfmxan 
A serve both planning and legislative 

requirements. ^ v ' - ; 

- ' • "• . * 

b. Lack of systematic,. flexible, and Well-defined 

written plans for vocational education evaluation 
systems. As a result, some states dre faced ttith 
communication problems snd Jack 7 of coordination 
between the different 'units of performing 
vocational education^ at the state level . v " , 

* ■ ' • • * " v" ' • * ' 

' \ v Re comme nd a tipns : 

The analysis of the state 1 s ability to evaluate its . 
programs and services for" special populations finally comes \ 
down to the capacity to generate certain units of data and 
information related to access, process,; participation, and - 
outcomes. This led the National , Center proje<5t staff to the 
conclusion that "evaluation is *|i^£6rm9tion^ i.e.., evaluation 
capability depends on the staters' ability to collect, store, . , 
analyze, compare, organize, and' recall information. 

> ' ■ ** • 

Thus, any state Which is pi inning for evaluation 
improvement should start by determining what ifk is they, and 
those* they are accountable to, will accept a^ fvidence of 
program effectiveness. Until this question is answered neither 
program operators or evaluators have any rational- basis upon 
which to proceed. . -'• * 



After is has been determined what the criteria of 
effectiveness are, the present data and information collection 
instruments and procedures should be examined to determine the 
extent to which they produce the needed information. 
Instruments and procedure? should then be revised or developed 
to fill in the gaps in the existing system* < 

* The presentation of data and information on program 
effectiveness should be viewed as a guide to action. This 
process should be constructed on the premise that certain 
programs are less than fully effective in specified measures 
and that .lack of effectiveness is due to certain conditions 
which should be corrected by specified actions*. 

Finally, implementing of the recommended actions and 
observations of the results completes the evaluation effort. 
(This entire process will be repeated several time.) 

Thus, the following recommendations are'given:* 

Criteria of effectiveness stated in measurable or 
observable program outcomes should be specified. 

Information .and data collecting and processing 
procedures should be reviewed and revised^to provide 
evidence of effectiveness* 

The best that is known about what causes quality in 
programs should be applied to those parts of the 
program which are not performing properly . 

Procedures for assisting programs to make the changes 
expected tp increase effectiveness should be 
established. 

Close observation should be maintained to assess the 
extent to which the steps taken above have resulted in 
improved program outcomes. 



i. 

2. 
3. 
4. 
5. 
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APPENDIX A 



STATE AGENCY EVALUATION COMPLIANCE REQUIREMENTS - 
TO FEDERAL LEGISLATION AND RULES AND REGULATIONS 
FOR SPECIAL POPULATIONS 



State personnel haye expressed concern with regard * to federal compliance for 
vocational education evaluation units of state agencies* The outline that follows 
presents items of concern which must be evaluated 1 to meet federal compliance 
requirements and includes referenced sources and actual citations as found in the law 
and rules and regulations of pertinent laws. 

Two things should be noted pertaining to compliance required as cited below: 

1) citations cited are those of direct concern to state agency evaluation units and 

2) state agency personnel are cautioned that compliance does not constitute the 
Meting of necessary criteria of an effective evaluation system* 



General Responsibilities of State Agencies 



Items 
1. General 



Authority 

Sec. 104.402 
(R & R of Ed. 
Amend, of 1976) 



2. Compliance 

Responsibilities 



>art 80. II-B 
(R & R of CP Reap. % 
fr: Title VI of the 
CRA of 1964, Title 
IX of the Ed*. Amend* 
of 1972, & Sec. 504 
of the R.A. of 1973) 



Citations 



The state board shall, during 
the five-year period of the 
state plan evaluate in quan- 
titative terms of effectiveness 
of each formally organized pro- 
gram or project supported by 
Federal, State and local funds 

The State agency responsible for 
the administration of vocational 
education programs must adopt a 
compliance program to prevent, 
identify and remedy discriraina- 1 
tion on the basis of race, color; 
national origin, sex or handicap^ 
by its subrecipients. (A "sutr 
recipient," in this context, is a 
local agency or vocational 
education center that receives 
financial assistance through a 
State agency.) 
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Items Authority Citatioiis 

i 



r 



This compliance program must 
include: 
1« Collecting and analyzing 
civil rights related data and 
information that subrecipients 
compile for their own purposes or 
that are submitted to State and 
Federal officials under existing 
authorities; 

2» Conducting. periodic ^ v 
compliance reviews of selected 
subrecipients (i.e., an 
investigation of a subrecipient* 
to determine whether it engages 
in unlawful discrimination, 
notifying* the subrecipient of 
steps it must take to attain 
compliance and attempting to 
obtain voluntary compliance; 
3* Providing technical 
assistance upon request to 5 
fit i subrecipients. This will include 

. assisting subrecipients identify 

unlawful discrimination and 
instructing them in remedies for 
an£ prevention of such 
discrimination; 

*4. Periodically reporting its 
activities and findings under the 
foregoing paragraphs, including 
findings of unlawful 
discrimination under paragraph .2$ 
immediately above, to the Office 
for Civil Right?. 



ACCESS 

Facility Access 
1.1 Definition 



84.3 



Subpart A Sec. 
CD • 

(R & R of P.L. 93-112 
as amended) 



(i) "Facility" means all or any 
portion of buildings, structures, 
equipment, roads, walks, parking 
lots, or other real or personal 
property or interest in such 
property c 
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Authority 



Citations 



Part 80-IV N(3) 
(R & R of CP Resp* 
fr: title VI of the 
CtfA of 1964; Title 
IX of the Ed. Amend, 
of 1972 & Sec. 504 
of the R.A. of 1973) 



Subpart A. Sec. 
84-4 

(R & R of P.L. 
93-112 as amended) 



Part 80. IV-B 
(R & R of CP Resp. 
fr: Title VI of Uhe 
CRA of 1964, Title 
IX of the Ed. Amend, 
of 1972 & Sec. 504 
of the R.A. of 1973) 



Recipients may not deny 
handicappped students access 
to vocational education programs 
or courses because of architec- 
tural or equipment barriers, or 
because of the need for related 
aids and services or auxiliary 
aids. 

In determining the site or 
location of a facility, an 
applicant for assistance or a 
recipient may not make , selection^ 
(i) that have the effect of * 
excluding handicapped persons 
from, denying them the benefits 
of , or otherwise subjecting them 
to discrimination under any 
program or activity that receives 
or benefits from Federal 
A financial assistance or (ii) that 
have , the purpose or effect of 
defeating or substantially 
impairing the accomplishment of 
the objectives of the program or 
activity with respect to 
handicapped persons. 

State and local recipients may 
not select or approve a site for, 
a vocational education facility 
for the purpose or with the * 
ejjjgtt of excluding, segregating, 
or^>therwise "discriminating 
against students on the basis of 
race, color, or national origin^ 
Recipients must locate vocational 
education facilities at sites 
that are readily accessible to 
both nonminority and minority 
communities, and that do not tend 
to identify the facility or 
program as intended for 
non-miiiority or minority 
students. 
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Items 

1.5 Modification 



1.6 Comparable 
facilities 



1.7 Housing 

opportunities 



Authority 

Part SO. IV-D 
£R & R of CR Resp. 
fr : Title VI of the 
CRA of 1964, Title 
IX of the Ed. Amend, 
of 1972, S^Sec. 504 
of the R. A, of 1973) 



Citations 

A recipient may not add to, 
modify, or renovate the physical 
plant of a Vocational education 
facility in a manlier that 
creates, maintains, or increases 
student segregation on the basis 
of race, color, national origin* 
sex, or handicap. . * 



Part 80. VI-D Recipients must provide changing 

(R & R of CR Resp. rooms, showers, and other 

fr: Title VI of the facilities for students of one . 

CRA of the 1964, Title sex that are comparable to those 

IX of the Ed. Amend. provided to students of the other 

v of 1972, & Sec. 504 sex. 
of the R.A. o^f 1973) { 

Part 80. VI-C 
(R & R of CR Resp. . 
fr: Title VI of the 
CRA of the 1964, 
Title IX of the Ed/ 
Amend, of 1972, & 
Sec. 504 of the R.A. 
of 1973) 



Recipients must extent housing 
opportunities ^without 
discrimination based on race, 
color, national origin, sex qr 
handicap. This obligation 
extends to recipients that 
provide on- campus housing and/ or 
that have agreements with 
providers of off-campus housing. 
In particular, a recipient 
post secondary vocational 
education program that provides 
on-campus or off-campus housing 
to its non-handicapped students 
must provide, at the same cost 
and under the same conditions, 
comparable convenient and 
accessible housing to handicapped 
students. 
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' Items 



Authority 



Citations 



Educational Access 
2.1 Recruitment 
/ 



J 



2.1.1 



Public 

notification 



Part 80 V-C 
(R'& R of CR Resp. 
fr: Title VI of the 
CRA of the 1964, 
Title IX of the Ed. 
Amend, of 1972, & 
Sec. 504 of the R.A. 
of 1973) 



/ 

Part 80. IV-D 

(R & R of CR Resp. 

fr: Title VI of the 

CRA of 1964, Title IX 

of the Ed. Amend, of . 

1972, & Sec. 504 

of the R.A, of 1973) 



0 



Recipients must condudt their 
student recruitment activities as 
not to exclude or limit 
opportunities *on the basis of 
race, color, national origin, 
sex, or handicap* Where 
recruitment activities involve 
the presentation or portrayal of 
vocational and career 
opportunities, the curricula and 
programs describied should cover a 
broad range of occupational 
opportunities and not be limited 
on the basis of the race, color, 
national org in, sex, or handicap 
of the students or potential 
students to whom the presentation 
is made. Also, to the extent 
possible, recruiting teams should 
include persons of different 
races, national origins, sexes, 
and handicaps* 

Prior to the beginning of each 
school year, recipients must 
advise students, parents, 
employees and the general public 
that all vocational opportunities 
will be offered without regard to 
race, color, national origin, 
sex, or handicap. Announcement 
of this policy of ^ 
non-discrimination may be made, 
for example, in local newspapers, 
recipient publications and/or 
other media that reach the ^ 
general public, prograib 
beneficiaries, minorities 
(including national origin 
minorities with limited English" 
language skills), worsen, and 
handicapped persons. A brief 
summary of program offerings and 
admission criteria, should be 
included in the announcement ;' 
also the name, address and 
telephone number of the person 
designated to coordinate Title IX 
"and Section 504 compliance 
activity. J . 
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Items 



2.1.2 Promotional 
efforts 



2.2 Admission criteria 



Authority 



V 



Part 80. V-E 
(R & R of CR Resp. 
fr: Title VI of the 
CRA of 1964, Title 
IX of the Ed. Amend • 
of 1972, & Sec. 504 
of the R.A. of 1973) 



Part 80. IV-A ^ 
(R & R of CP Resp. 
fr: Title $1 of the 
CRA of 1964, Title 
JX of the Ed. Amend, 
of 1972, & Sec. 504 
of the R.A. of 1973) 



Citations: 

If a recipient's service con- 
tains a community of national 
origin minority persons with 
limited English language skills, 
public notification materials 
must be disseminated tti that 
community in its language and 
must state that recipients will 
take steps to assure that the 
lack of English language skills 
will not be a barrier to 
admission and participation in 
vocational education programs. 

Recipients^ must not undertake 
promotional efforts (including 
activities of school officials, 
counselors, and vocational staff) 
in a manner that creates or 
perpetuates stereotypes or 
limitations based on race, color, 
national origin, sex or handicap. 

Criteria controlling student 
eligibility for admission to 
vocational education schools, 
facilities and programs may not 
unlawfully discriminate on the , 
basis of race, color, national 
origin, sex, or handicap* A 
recipient may not develop, 
impose, maintain, approve, or 
implement such discriminatory 
admissions criteria. 
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Items 



Authority 



Citations 



2.2.1 



Eligibility 
based on 
residence 



2.2.2 



Eligibility 
based on 
numerical 
limits 
(secondary) 



Part 80. IV-C 
(R & R of CR Resp. 
fr: Title VI of the 
CRA of 1964, Title 
IX of the Ed. Mend, 
of 1972, & Sec. 504 
of the R.A. of 1973) 



Part 80. IV-F 
(R & R of CR Resp. 
fr: Title VI of the 
CRA of 1964, Title / 
IX of the Ed. Amend, 
of 1972, & Sec. 504 
of the R.A. of 1973) 



Recipients o may not establish, 
approve or maintain geographic *" 
boundaries for a vocational 
education center service area^or 
attendance zone, (hereinafter 
w service area" ) , that unlawfully 
exclude students on the basis of 
race, color, or national origin. 

A recipient may not adopt or 
maintain a system for admission 
to a secondary vocational 
education center or program that 
limits admission to a fixed , 
number of students from each \ 
sending school included in the 
center's service ar^lLf such a 
system disproportioj^fcly 
excludes students j^Bn the center 
on the basis of radBf sex, 
national origin, or%andicap. 
(Example: Assume 25 percent of a 
school district's high school ^ 
students are black and that most 
of those black students are 
enrolled in one high school; the 
white students, 75 percent of the 
district's total enrollment, are 
generally enrolled in 'the five 
remaining high schools. This 
paragraph prohibits a system of 
admission to the secondary 
vocational education center that 
limits eligibility to a fixed and 
equal number of students from 
each of the district's six high 
schools.) 
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Items 



Authority 



Citations 



2*2.3 Eligibility , 
baked upon 
student v 
option < race, 
national ori- 
gin and sex) 



2.2.4 migibility 
based on ap- 
plicant 
evaluation 



2.2.5 Eligibility 
based on 
language 



Part 80. IV-H 
(R & £ of CR Resp* 
fr: Title VI of the 
CRA of 1964. Title 
IX of the Ed. Amend. 
9f 1972, & Sec. 504 
of the R.A. of 1976) 



Part 80. VI-K 
(R & R of CR Resp. 
fr: Title VI of the , 
CRA of 1964, *Title 
IX of the Ed. Amend, 
of 1972, & Sec. 504 
of the R.AV of L973) 



Part 80. IV-L 
(R & R of CR Resp. 
fr; Title VI of the 
CRA of 1964, Title 
IX of the Ed. Amend, 
of 1972, & Sec. 504 
of the R.A. of 1973) 



A vocational education center, 
branch, or annex, oj>en to all 
students in a service area and 
predominently enrolling minority 
students or students of one race , 
national origin or sex, will be 
• presumed unlawfully segregated 
if: 1) it was established by a 
recipient f§r members of one 
race, national origin ox; sex; or 
2) it has since »it& construction 
been attended primarily by 
members of one race, national 
origin or sex; or 3) most of its 
program offerings have* 
traditionally been selected 
predominently by members of one 
race, national origin or sex. 

Recipients may not judge 
candidates for admission to 
vocational education programs on 
the basis of criteria that have 
the effect* of disproportionately 
excluding persons of a particular 
race, color, national origin, 
sex, or handicap. w 

Recipients may not restrict an 
. applicant f s admission to 
vocational education programs 
because the application, as a 
member of a national origin 
minority with limited English 
language skills , cannot 
participate in and benef it * f rom 
vocational instruction to the 
same /extent as a student whose 
primary language is English. It 
is the responsibility of the 
recipient to identify such 
applicants and access itheir 
ability to participate in 
vocational education. 
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Items 



2.2.6 Access basec 
on eApl e y niCTtt 
opportunities 
(handicapped 
only) .<* 



2.2*7 Eligibility 
based on age 



Authority 

Part 80. IV-N 
(R & R of CR Resp. 
fr: Title VI of the 
CRA of 1964, Title 
IX of the Ed. Amend, 
of 1972, & Sec. 504 
of the, R. A. of 1973) 



Public Law 94-135 
Title Ill-Sec. 303 



Citations 



B. PARTICIPATION 

1. Enrollment of special 
populations 



Sec.M04.116 
(R & R of CR Resp. 
fr: Title VI of ■ the 
CRA of 1964, Title 
IX of the Ed. Amend, 
of 1972, & Sec. 504 
of the R.A. of 1973) 



r 



ia^i 



Access to vocational programs or 
courses may not be denied 
handicapped students on the 
ground that employment 
opportunities in any occupation 
« tfr profession may be more limited 
for handicapped persons than/for 
non-handicapped persons. . 

• .. • • • ts 

It is the purpose of this title 
prohibit unreasonable 
discrimination on the basis of 
age in programs and activities 
Receiving Federal assistance, 
including programs in activities 
receiving funds under the State 
and Local Fiscal Assistance Act 
of 1972 (31,U.S;C. 1221 et seq.) 



(a) The Commissioner and the 
Administrator of NOES will 
jointly develop information 
elements and uniform definitions 
fbr a national educaiton data 
reporting and accounting system. 

(b) This systenuwill include 
infdrmation resulting v frdm the 
evaluations under section 112(b) 
of the Act (Sees. 104.40? and 
104.404) and other information on 
vocational: 

(1) Students (including 
information on their race and. 
sex); . 

(2f) Programs; 

(3 ^Program completers and 
leavers; 

(4) Staff; 

(5) Facilities; and 

(6) Expenditures 
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Items • Authority 

PROGRESS AND D. OUTCOMES 



General responsibility 



Sec* 104*402 
(R & R of P*L* 
93-112 as amended) 




4 



Citations 



The State board shall, during the 
five-year period of the State 
plan, evaluate in quantitative 
terms the effectiveness of each 
formally organized program or > 
project supported by Federal, 
State, and local funds* These 
evaluations shall be in terms of: 

(a) Planning and dpefatiqnal 
processes, such as 

(1) Quality and availability of 
instructional offerings; 

(2 ) Guidance , -counseling , and 
placement and follow-up services; 

(3) Capacity and condition of 
facilities and equipment;. 

(4) Employer prticipation in 
cooperative programs of 
vocational education; 

(5) Teacher/pupil ratios; and 

(6) Teacher qualifications, 

(b) Results of student ( 
achievement as measured, for 
example , by; 

(1) Standard occupational 
proficiency measures; 

(2) Criterion reSerenced tests; 
and 

Other examinatibns of 
students* skills^ knowledge, 
attitudes, and re&diness for 
entering employment successfully* 

(c) Results of student employment 
success as measured, for example, 
by: 

(1) Rates of employment and 
unemployment; 

(2) Wage rates; ■ 

(3) Duration of employment; and 

(4) Employer satisfaction with 
performance of* vocational 
education students as compared 
with performance df persons who 
have not had vocational 
education* 
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It< 



Authority 



Citations • yf 



Sex stereotyping 



Sec. 104.75(e) 
(R & R of Ed. Amend, 
.of 1976.) 



(d) The result s^bf additional 
services, as measured by the 
suggested criteria under 
paragraphs (a), (b), and (c) of 
this section, that the State 
provides under the Act to these- 
special populations ; 

(1) Women} 

(2) Members of minority groups; 
j(3) Handicapped persons; 

(4) Disadvantaged persons; and 

(5) Persons of limited 
English-speaking 



Review all vocational education 
programs (including worle-study 
programs, cooperative vocational 
education programs, 
apprenticeship programs , ind 
placement of students vho have 
successfully completed vocational 
education programs) in State for 
sex bias • 



^ Counseling materials 



Part 80. V-A 
(R & R of CR Resp. 
fr: Title VI of the 
CRA^)f 1964, Titles 
IX of the Ed . Amend . 
of 1972, & Sec. 
of the R.A. of A 1973) 



Recipients must insure that their 
counseling materials &nd 
activities (including student ' 
program selection and 
career/ employment flection) , 
-promotional , ^and^recruitment 
if forts do not discriminate on 
the basis of race, color, 
national origin^s^x or handicap. 



Counselors responsi- 
bility (LESA & Hearing 
Impairment) 



Part 80. V-D 
(R & R of CR Resp. 
fr; Title VI of the 
CRA of 1964, Title 
IX of £he Ed. Amend, 
of 1972, & Sec. 504 
of the R.A. of 1973) 



Recipients must insure that 
counselors can effectively 
communicate with national origin 
minority students with limited 
English language skills and with 
Students who have hearing 
impairments • This requirement s 
may be satisfied by having 
interpreters available. 
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Items 



Authority 



Citations 



Work Study, Cooperative Part 80. VII-A 



Vocational Education, 
Job Placement, and * 
Apprentice Training 



Accountability Report 



IE# (Handicapped) 



(R & R of CR Res p. 
fr : Title VI of the 
CRA of 1964, Title 
IX of the Ed. Amend, 
of 1972, & Sec. 504 
of the R.A. of 1973 



Sec. 104.241 (a)(1) 
(R & R of Ed. Amend, 
of 1976) 



Sec. 121.a34(a) 



A recipient must insure that (a) 
it does not discriminate against . 
its students on tKe basis of 
race 9 color, national origin, » 
sex, or handicap in making 
available opportunities in 
cooperative education, work study 
and job placement programs, and 
(b) st^jtefiE^ participating in 
cooperative education, work study 
and job placement programs are % 
not discriminated against by : 
employers or prospective - 
employers on the basis of race, 
color, national origin, sex, or 
handicap in recruitment, hiring, 
placement, assignment to work 
tasks, hours of employment, 
levels of responsibility, and in 
pay . _ 

Show the extent to which the 
Statfe , ^during the fiscal year 
preceding the submission 'of the 
report, has achieved the goals of 
the approved five-year state 
plan, including a description in 
terms of the elements of Sec* 
,104.184. 

The State educational agency 
shall insure that each public 
agency develops and implements- an 
individualized education program 
for each of its handicapped 
children. 
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APPENDIX B 

EVALUATION OF PROGRAMS AND SERVICES 
FOR SPECIAL POPULATIONS 

SELF-ASSESSMENT INSTRUMENT ■• • . » 

Objectives: To access the states ability to: 

1. Determine accessibility of programs for special populations 
^2. Identify services being provided for special populations 

3. Determine participation of special populations in programs 

4. Measure outcomes of programs and services for special populations 



Direction: Kindly check the appropriate space of each criterion of Interest 
using the rating scale below. 



Needs Major Improvement 



Needs Minor Improvement 



Satisfactory 



Conditions are below 
standards; needs 
additional emphasis 

Conditions are meeting 
standards; could stand 
minor improvement 

Conditions exceed minimum 
standards; practically 
needs little or no 
improvement 
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I: Ability to Determine Accessibility of Programs for Special Populations 



/ 



Criteria 

THROUGH ITS VOCATIONAL EVALUATION SYSTEM, THE STATE IS ABLE TO 
DETERMINE ACCESSIBILITY OF VOCATIONAL PROGRAMS FOR SPECIAL 
POPULATIONS AS INDICATED BY; 



1. Periodic review of facility factors that tend to 
increase accessibility of vocational^ pro-ams. 

Evidence: The evaluation instrument(s) and evaluation 



4 



O 



2. 



eport(s) include investigation of the facility 
factors specified by DHEW, Federkl Register Vol. 44, 
No. 56, March 21, 1979: 

1.1 Architectural and equipment 

1.2 Site location (non-discriminatory) 

1.3 Site selection (non-discriminatory) 

1.4 Modification of physical plant 

1.5 Comparable facilities 

1.6 Housing opportunities 
1*7 Topographical 

Periodic review of school policies/practices that tend 
to increase accessibility of vocational programs. These 
are: 



Years 



v 



Ratings 



79-80 80-81 81-82 82-83 



lou 



2. 1 Recruitment procedures employed at the local school 
level ^ 

Evidence ? The evaluation instrument (s) and evalua- 
tion report(s) include investigation of the following 
recruitment procedures ; 

2.1.1 
2.1.2 
2.1.3 
2.1.4 



4 



2.1.5 



Public notification 
Promotional efforts 

Identification of potential students in school system 
Identification of potential students outside of 
the school system -"~ 
Parent motivation 
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2.2 Local admlsalon criteria for special populations 
to enter Into vocational programs 

Evidence ! The evaluation instrument s) and evaluation 
reportCs) Include Investigation of the following discrim- 
inatory practices to avoid as specified by DHEW, Federal 
Register Vol. 44, No. 56, March 21, 1979: 

2.2.1 Eligibility based on residence 

2.2.2 Eligibility based on numerical limits 

2.2.3 Eligibility based on student option (race, 
national origin and sex designation) 

2.2.4 Eligibility based on applicant evaluation 

2.2.5 Eligibility based on language 

2.2.6 Access based oh employment opportunities 
. (handicapped) 

2.2.7 Jligibility based on age 

2.3 Diversity/ adequacy of local program offerings. 

Evidence ! The evaluation instrument^ s) used and 
evaluation report (s) include investigation of the 
following factors! 

2.3.1 Adequacy of educational opportunities 

2.3.2 Diversity of program offerings 

2.3.3 Trained staff 

2.3.4 Modified course presentation (handicapped) 

2.3.5 Support services 

a* day care ^ 
b. auxiliary aids 

el remedial ' 
d. financial 

2.3.6 Program comprehensiveness 

- job, social, and employ ability skills 



Ratings 
f 3 



79-80 



Years 
80-81 



81-82 



82-83 



9 
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Years 



Ratings 79-80 80-81 81-82 82-83 

3* Periodic review of local school efforts to remove: 

3*1 Attltudinal barriers 3 s % \ ... 

Evidence: The evaluation instrument(s) and evaluation 2 ______ ' • "■_ . 



to 



Evidence: A written state plan* 



2 
1 



report (s) include investigation^of local school efforts 

to remove attltudinal barriers such as: 1 .' 

3* 1* 1 Stereotyping person's ability to do work 

3*1*2 Personal negative self-image , 

3.1*3 Job stereotyping 

3*1*4 Nonraccepting attitudes in society . 4 . 

3*2 Societal barriers including the promotion of 3 . 

equal opportunity in employment and promotion 

/ 2 _ 

Evidence : The evaluation instrument(s) and eval- 9 

uation report(s) include investigation of local school 1 

£ / efforts to remove other societal barriers such as: 

3. 2.1 Discriminatory practices in hiring * 

3.2*2 Discriminatory practices in salary 

3.2*3 Discriminatory practices in promotion 

3*2*4 Discriminatory practices in public services 

4* The states has a well-defined system to increase 3 \ 

accessibility pf vocational programs* 
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,IIp Ability to Determine Participation of Special Populations in Vocational Programs ^ 

Years 
79-80* 80-81 



( 



/ 



f Criteria . j _ . 

1* The state has a complete record of vocational program 
enrollment of the different categories of special popu- 
lations and each category is further subdivided according 
to VEDS classification . " 

Evidence : State summary enrollment of special popula- 
tion enrollment in all vocational programs has the following 
data: 



Ratings 
3 
2 
1 



81-82 



82-83 



VEDS Classification: 

1.1 Disadvantaged 

1.1.1 Economically 

1.1.2 Academical ly 

1.2 Handicappedj 

1.2.1 Mehtally retarded 

1.2.2 Hard of hearing 

1.2.3 Deaf 

1.2.4 Speech impaired 

1.2.5 Visually handicapped 

1.2.6 Emotionally disturbed 

1.2.7 Orthopedically impaired 

1.2.8 Other health impaired 

1.2.9 Specific learning disabled 



1.3 



1.4 



1.5 



Minorities , 

1.3.1 American Indian/ Alaskian Native 
1.-3.2 Asian Am6ricai^Pacific Islander 
1.%.3 Blacky not Hispanic 

1.3.4 /Ttiepanic 

1.3.5 White, not Hispanic 

Limited English Speaking 

1.4.1 Spanish dialect 

1.4.2 Italian dialect 

1.4.3 Other 

Sex Designation > . 9 ' 

1.5.1 Female 

1.5.2 Male 
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Criteria 

2. The state has a complete record of enrollment of 
special populations by program . 

Evidence : State summary special population enrollment 
data by USOE 6-digit and USOE 2-digit codes. 



3. The stater has a complete record of handicapped 
enrollment as to instructional settings . 

Evidence ; Handicapped data as to instructional 
settings: 

Instructional Setting 
' 3. 1 regular class 

3.2 mixed class 

3.3 separate class ' 

3.4 separate facility 

3.5 other 



4. The state has a well-defined system of determining 
participation of special^ population in vocational 
programs. 

** ■ . • 

/Evidence: A well defined plan or reports of 
enrollment. 



3 
2 
1 



Years . 

Ratings 79-80 80-81 , 81-82 82-83 

3 ' V . : 

2 • ■ : " . : '•'•••■„" . 

1 ; ■"' ■ : 



3 
2 
1 



1 
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III. Ability to Identify Services for Special Populations 



Criteria 

THROUGH ITS VOCATIONAL EVALUATION SYSTEM, THE STATE IS 
ABLE TO IDENTIFY SERVICES FOR SPECIAL POPULATIONS AS 
INDICATED BY: 



Ratings 



79-80 



Years 
80-81 



81-82 82-fc3 



Periodic review of the quality of instructional offerings 
for special populations including ^tifhe. capacity and condi- 
tion of facilities and equipment for special populations 

and the use of bias-free materials* 

* ■ ■ ■ 

Evidence : Inclusion of instruction related factors V* 
in the evaluation instrument (s) and a separate 
analysis and report on the subject in the evaluation 
reports.- 



3 
2 
1 



Periodic review of the quality of guidance and counseling 
for special populations including a review of possible 
discriminatory counseling materials and activities. 

Evidence : Guidance and counseling for special populations 
in the total evaluation system are included i^the evaluation 
ins t rumen t(s) and evaluation rfepo£t(s). 



3 
2 




Criteria 



Periodic review of the quality of placement services 
for special populations including a review of possible 
employer or prospective employer discrimination. 

Evidence : Placement service for special population 
is included in the evaluation instrument(s) and evalua- 
tion report(s). 



Periodic review of other services 

Evidence : Other services for special populations are 
shown included in the evaluation instrument^ s) and 
evaluation report(s). - 



A well defined system to identify servicer for special 
populations . ^ 

» 

Evidence ; State evaluation pldn includes identification 
of services for special populations 



16, 



i 



^ears 

Ratings 79-80 80-81 81-82 82-83 



3 
2 
1 




7 



Rajtings 
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IV. Ability to Measure Program Outcomes for Special Populations 

^Criteria 

1. The state has datafand information on the special 
population regarding the following: 

1.1 Successful Program Completions { 3 
Evidence : Follow-up report has the^ following data: 2 
Successful Program Completion: 1 

1.1.1 Grades 11 and 12 

1.1.2 Post secondary 
1.1.3, Adult 

1.1.4 Apprenticeship 

1.2 Student Placement 3 

Evidence : Follow-up report has the following data: 2 

Student Placement (completers/leavers-racial/ethnic/sex 1 

designation and handicapped): 



79-80 



Years 
80-81 



81-82 82-83 



1.2.1 Employed 

- in field related to training 

a. civilian M 

b. military 

in field not related to training, 

a. civilian 

b. military 

- salary rates 

- employer satisfaction 

- others 



1.2.2 Unemployed . m 

- seeking employment and not pursuing additional 
education , . 

-^not in labor force and not pursuing additional 
education 

1.2.3 Pursuing Additional Education 

1.2.4 Status Unknown 



ERLC 
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Ratings 

1.3 Succga^ul Employment Over Time 3 

Evidence: The follow-up report has the following data: . 2 

Successful Employment Over Time 1 
1*3.1 Duration 

1.3.2 Promotions y 

1.3.3 Salary Increases 

1.3.4 Reactions to Training ? ^ , 



79-80 



Years . 
80^81 



81-82 82-83 



00 



2. The state has the ability to measure student 

achievement of special populations with standardized - 
occupational proficiency measures, criterion- 
referenced tests, etc. 

Evidence : State data showing results of student 
achievement are measured by standard proficiency 
measures, criterion-referenced tests, etc. 
* Standardized Instrument to measure student ( achievement. 
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